[Python] pyarrow=13 requires pandas installed to use `to_pandas_dtype()` (which is usually a numpy datatype)

### Describe the bug, including details regarding any error messages, version, and platform.

In pyarrow 12 and earlier (back to at least 5), the following would work just fine without pandas installed (conda forge env with `python=3.11 pyarrow=12`):

```
import pyarrow as pa
dt = pa.int64()
dt.to_pandas_dtype()
```
which outputs `numpy.int64` as expected.

However, with pyarrow 13 this happens:
```
ModuleNotFoundError Traceback (most recent call last)
Cell In[4], line 1
----> 1 dt.to_pandas_dtype()
File ~/miniforge3/envs/arrowpand13/lib/python3.11/site-packages/pyarrow/types.pxi:326, in pyarrow.lib.DataType.to_pandas_dtype()
File ~/miniforge3/envs/arrowpand13/lib/python3.11/site-packages/pyarrow/types.pxi:153, in pyarrow.lib._to_pandas_dtype()
File ~/miniforge3/envs/arrowpand13/lib/python3.11/site-packages/pyarrow/pandas-shim.pxi:158, in pyarrow.lib._PandasAPIShim.is_v1()
File ~/miniforge3/envs/arrowpand13/lib/python3.11/site-packages/pyarrow/pandas-shim.pxi:104, in pyarrow.lib._PandasAPIShim._check_import()
File ~/miniforge3/envs/arrowpand13/lib/python3.11/site-packages/pyarrow/pandas-shim.pxi:53, in pyarrow.lib._PandasAPIShim._import_pandas()
File ~/miniforge3/envs/arrowpand13/lib/python3.11/site-packages/pyarrow/pandas-shim.pxi:48, in pyarrow.lib._PandasAPIShim._import_pandas()

ModuleNotFoundError: No module named 'pandas'
```

This is a serious regression. I use pyarrow so that I can access parquet and arrow table functionality _without_ pandas. And there's no API for `to_numpy_dtype()` (which I know has been asked for).

### Component(s)

Python