[BUG] ParquetDatasetWriter(max_file_size=...) leaks file descriptors across rotations
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
## Describe the bug
With `ParquetDatasetWriter(max_file_size=...)`, each file rotation creates a new `ParquetWriter`, but retired writers are retained in `self._chunked_writers` until the final `close()`. On a workload with 1,000 partitions this exhausts a 131,072-FD limit during the second append.
This is distinct from [#23378](https://github.com/rapidsai/cudf/issues/23378): that issue is about the size estimator causing premature rotation; this issue is that rotated writers are not closed/released.
## Reproduction
The full reproducer and output are in [VibhuJawa/cudf#10](https://github.com/VibhuJawa/cudf/issues/10). The essential setup is:
```python
from cudf.io.parquet import ParquetDatasetWriter
w = ParquetDatasetWriter(
"/tmp/pdw_fd_leak",
partition_cols=["centroid"],
index=False,
max_file_size="20MB",
file_name_prefix="part",
)
for i in range(4):
w.write_table(make_df(i)) # 500k rows, list[1024],
# 1,000 centroid values
w.close()
```
On cuDF `26.08.00a990`, H100 80GB, Python 3.12, CUDA 12.9, and `RLIMIT_NOFILE=131072`:
```
start: 5 FDs
after append 0: 126,038 FDs
append 1: RuntimeError: Unable to open file ... Too many open files
```
The same workload without `max_file_size` stays at about 2,038 FDs and completes. The result reproduced on node-local ext3 and WekaFS.
## Source indication
The current implementation appends a new `ParquetWriter` at [python/cudf/cudf/io/parquet.py#L2265-L2277](https://github.com/rapidsai/cudf/blob/main/python/cudf/cudf/io/parquet.py#L2265-L2277), while [close()](https://github.com/rapidsai/cudf/blob/main/python/cudf/cudf/io/parquet.py#L2279-L2289) closes every writer only at teardown.
## Expected behavior
Retired chunked writers should be closed as soon as their paths are rotated out, keeping descriptor usage bounded by the active partitions. `close()` should also release all remaining descriptors.
## Environment
- cuDF `26.08.00a990` nightly; not yet checked on a released build
- H100 80GB, CUDA 12.9, Python 3.12
- `RLIMIT_NOFILE` soft=hard=131,072
Contributor guide
Research direction
Start in python/cudf/cudf/io/parquet.py around lines 2265-2289 and run the provided multi-partition reproducer to observe descriptor growth across appends. Trace how rotated writers are stored and how close() handles them. Done means rotated descriptors are released promptly, remaining writers are released by close(), and the workload completes without exhausting file descriptors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100