`pyarrow.infer_type` leaks memory with datetime arrays

### Describe the bug, including details regarding any error messages, version, and platform.

Two issues:
1. `pa.infer_type` uses something like O(n^2) memory to infer the type
2. The memory isn't cleaned up

```python
import gc

import numpy as np
import psutil
import pyarrow as pa

array = np.zeros(8 * 1024 * 1024 // 8, dtype="datetime64[s]") # A 8 MiB array
process = psutil.Process()

for _ in range(30):
before = process.memory_info().rss

pa.infer_type(array)

gc.collect()
pa.default_memory_pool().release_unused()
after = process.memory_info().rss

print(
f"{(after - before) / 1024 / 1024:.02f} MiB ({before / 1024 / 1024:.02f} MiB "
f"-> {after / 1024 / 1024:.02f} MiB)"
)
```

```
190.53 MiB (46.95 MiB -> 237.48 MiB)
146.28 MiB (237.50 MiB -> 383.78 MiB)
146.33 MiB (383.78 MiB -> 530.11 MiB)
146.23 MiB (530.11 MiB -> 676.34 MiB)
146.30 MiB (676.34 MiB -> 822.64 MiB)
146.30 MiB (822.64 MiB -> 968.94 MiB)
146.31 MiB (968.94 MiB -> 1115.25 MiB)
146.30 MiB (1115.25 MiB -> 1261.55 MiB)
146.28 MiB (1261.55 MiB -> 1407.83 MiB)
146.30 MiB (1407.83 MiB -> 1554.12 MiB)
146.69 MiB (1554.12 MiB -> 1700.81 MiB)
146.30 MiB (1700.81 MiB -> 1847.11 MiB)
146.25 MiB (1847.11 MiB -> 1993.36 MiB)
146.33 MiB (1993.36 MiB -> 2139.69 MiB)
146.30 MiB (2139.69 MiB -> 2285.98 MiB)
146.30 MiB (2285.98 MiB -> 2432.28 MiB)
146.30 MiB (2432.28 MiB -> 2578.58 MiB)
146.30 MiB (2578.58 MiB -> 2724.88 MiB)
146.25 MiB (2724.88 MiB -> 2871.12 MiB)
146.30 MiB (2871.12 MiB -> 3017.42 MiB)
146.70 MiB (3017.42 MiB -> 3164.12 MiB)
146.34 MiB (3164.12 MiB -> 3310.47 MiB)
146.25 MiB (3310.47 MiB -> 3456.72 MiB)
146.30 MiB (3456.72 MiB -> 3603.02 MiB)
146.30 MiB (3603.02 MiB -> 3749.31 MiB)
146.27 MiB (3749.31 MiB -> 3895.58 MiB)
146.30 MiB (3895.58 MiB -> 4041.88 MiB)
146.31 MiB (4041.88 MiB -> 4188.19 MiB)
146.31 MiB (4188.19 MiB -> 4334.50 MiB)
146.28 MiB (4334.50 MiB -> 4480.78 MiB)
```

### Component(s)

Python