[C++][Python][Parquet] Duplicate metadata memory allocation when creating FileSystemDataset from _metadata file
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
**Describe the bug**
When creating a FileSystemDataset using pyarrow.dataset.parquet_dataset with a metadata file, the memory consumption is approximately double the expected amount due to duplicate metadata being loaded and retained in memory.
**Environment**
System: Amazon Linux 2023
Python: 3.9.16
PyArrow: 20.0.0
Dataset Characteristics:
- Metadata file size: ~200MB
- Contains hundreds of file fragments
- Row groups sized between 1e6-1e8 rows
**Run Details:**
- Python file:
```
import pyarrow.dataset as ds
myds = ds.parquet_dataset("/home/ec2-user/tmp/dataset/key1=value1/key2=value/_metadata")
```
- Command
`valgrind --tool=massif --pages-as-heap=yes --time-unit=ms python3 mem_analysis.py`
- Output
```
==49947== Massif, a heap profiler
==49947== Copyright (C) 2003-2017, and GNU GPL'd, by Nicholas Nethercote
==49947== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==49947== Command: python3 /home/ec2-user/tmp/pricepath_memanalysis/mem_analysis.py
==49947==
==49947== brk segment overflow in thread #1: can't grow to 0x4845000
==49947== (see section Limitations in user manual)
==49947== NOTE: further instances of this message will not be shown
/home/ec2-user/.local/lib/python3.9/site-packages/numpy/_core/getlimits.py:545: UserWarning: Signature b'\x00\xd0\xcc\xcc\xcc\xcc\xcc\xcc\xfb\xbf\x00\x00\x00\x00\x00\x00' for does not match any known type: falling back to type probe function.
This warnings indicates broken support for the dtype!
machar = _get_machar(dtype)
==49947==
```
- Generate ms_print output:
`ms_print massif.out.49947 > ms_print.49947`
**Expected behavior**
Declaring the dataset using `pyarrow.dataset.dataset` and calling `count_rows` over it caches the fragments metadata. So, the memory usage of `pyarrow.dataset.parquet_dataset` should have been similar.
**Actual Behavior**
Valgrind massif output shows duplicate memory allocations for metadata, totaling approximately double the expected memory consumption. Attaching the output of the `ms_print` command as screenshot and as file.

[ms_print.49947.txt](https://github.com/user-attachments/files/20077031/ms_print.49947.txt)
**My Analysis**
- `ParquetDatasetFactory::Make()` loads the metadata file and creates a factory object
- `factory.finish()` line in Python calls `ParquetDatasetFactory::CollectParquetFragments()`
which creates subsets of metadata via `FileMetaData::FileMetaDataImpl::Subset()`
- The original factory object's metadata remains allocated after `finish()` which causes duplicate memory
- My thinking is that the factory's `filesystem_` and `format_` members (held as shared pointers) are passed into the `FileSystemDataset` constructor during the `Finish()` call. This shared ownership could potentially prevent the factory object's timely de-allocation, including its metadata.
### Component(s)
Python, Parquet, C++
Contributor guide
Assessment
This issue has not been assessed yet.