Memory leak in Table indices

### Description

Repeatedly accessing an indexed `Table` causes memory use to grow in an unexpected and undesired way. In a real-world application on a large table this was causing memory use to exceed 18 Gb. Removing the table index and repeating the access code kept memory use below 1 Gb. We used `memray` to see memory climbing continuously during a loop which repeatedly accessed elements of an indexed table.

### Expected behavior

Memory use should remain approximately constant after the first access.

### How to Reproduce

The following should reproduce the problem. You can use a package like `memray` to monitor memory or just watch a process monitor for memory use of the Python process. For me this starts with about 180 Mb of memory after the first table `t` is created. After running this memory use is around 1 Gb, while I would expect something under 400 Mb.

```python
import numpy as np
from astropy.table import MaskedColumn
from astropy.table.table_helpers import simple_table
from astropy.time import Time
from tqdm import tqdm

size = 250000
t = simple_table(size=size, cols=26)
idxs = Time(np.random.randint(0, size // 20, size=size), format="cxcsec").isot
t["idx"] = MaskedColumn(idxs) # THIS IS THE PROBLEM
t.add_index(["idx"])

idxs = np.random.choice(t["idx"], size=100, replace=False)
for idx in tqdm(idxs):
star_obs = t[t["idx"] == idx]
```

### Versions

```python
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import astropy; print("astropy", astropy.__version__)
import numpy; print("Numpy", numpy.__version__)
import erfa; print("pyerfa", erfa.__version__)
try:
import scipy
print("Scipy", scipy.__version__)
except ImportError:
print("Scipy not installed")
try:
import matplotlib
print("Matplotlib", matplotlib.__version__)
except ImportError:
print("Matplotlib not installed")
```
```
macOS-14.2.1-x86_64-i386-64bit
Python 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:27:35) [Clang 14.0.6 ]
astropy 5.3.1
Numpy 1.23.5
pyerfa 2.0.0.1
Scipy 1.10.0
Matplotlib 3.6.3
```