[Python] Table.to_pandas Can't convert date32 to "datetime64[ns]"

### Describe the bug, including details regarding any error messages, version, and platform.

This is a follow up from this issue https://github.com/apache/arrow/issues/37545

I'm trying to convert arrow data of type `pyarrow.date32()` to pandas, and would like these date to be converted to `datetime64[ns]`. To do so I provide a `types_mapper` argument: `{pa.date32(): "datetime64[ns]"}`

I've noticed it is work with `pa.Array`, but not with `pa.Table`:
```python
import datetime

import pyarrow as pa
import pytest

date_array = pa.array([datetime.date(2023, 1, 1)], pa.date32())

assert (
date_array.to_pandas(types_mapper={pa.date32(): "datetime64[ns]"}.get).dtype
== "datetime64[ns]"
)

table = pa.Table.from_arrays([date_array], ["date"])

with pytest.raises(
ValueError,
match=r"This column does not support to be converted to a pandas ExtensionArray",
):
table.to_pandas(
types_mapper={pa.date32(): "datetime64[ns]"}.get
).dtype == "datetime64[ns]"

```

I'm would expect the behaviour to be consistent between pa.Array and pa.Table. I'm happy to submit an MR to fix the bug if pointed in the right direction.

I'm on
- pandas==2.1.2
- pyarrow==14.0.1

### Component(s)

Python