Memory leak in `fragment.to_table`

This "pseudo" code ends with OOM.

 
```java

import fsspec
import pyarrow
import pyarrow.parquet as pq

fs = fsspec.filesystem(
    "s3",
    default_cache_type="none",
    default_fill_cache=False,
    **our_storage_options,
)
dataset = pq.ParquetDataset(
    "path in bucket",
    filesystem=fs,
    filters=some_filters,
    use_legacy_dataset=False,
)

# this ends with OOM
dataset.read(columns=columns_to_read)

# and this too
tables = []
for fragment in dataset.fragments:
tables.append(fragment.to_table(columns=columns_to_read))
all_data = pyarrow.lib.concat_tables(tables)
```
What is really weird is if we put a debug point in the loop and **load** just {**}one fragment{**}. It loads, but something **keeps eating memory after load** until there is no left.

We are trying to read a parquet table that has several files under desired partitions. Each fragment has tens of columns and tens of millions of rows.

**Reporter**: [ondrej metelka](https://issues.apache.org/jira/browse/ARROW-16028)

_{**Note**: *This issue was originally created as [ARROW-16028](https://issues.apache.org/jira/browse/ARROW-16028). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*}

Start by reproducing the memory growth with the provided Python examples, focusing on ParquetDataset.read and fragment.to_table. Trace the relevant PyArrow dataset and Parquet-reading entry points; done means reading the fragments no longer continues consuming memory until the process reaches OOM.

Written by the indexing model from the issue text.