NVIDIA / NVIDIA/cudf

[BUG] ParquetDatasetWriter(max_file_size=...) leaks file descriptors across rotations

Open
#23,501 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

## Describe the bug

With `ParquetDatasetWriter(max_file_size=...)`, each file rotation creates a new `ParquetWriter`, but retired writers are retained in `self._chunked_writers` until the final `close()`. On a workload with 1,000 partitions this exhausts a 131,072-FD limit during the second append.

This is distinct from [#23378](https://github.com/rapidsai/cudf/issues/23378): that issue is about the size estimator causing premature rotation; this issue is that rotated writers are not closed/released.

## Reproduction

The full reproducer and output are in [VibhuJawa/cudf#10](https://github.com/VibhuJawa/cudf/issues/10). The essential setup is:

```python
from cudf.io.parquet import ParquetDatasetWriter

w = ParquetDatasetWriter(
"/tmp/pdw_fd_leak",
partition_cols=["centroid"],
index=False,
max_file_size="20MB",
file_name_prefix="part",
)
for i in range(4):
w.write_table(make_df(i)) # 500k rows, list[1024],
# 1,000 centroid values
w.close()
```

On cuDF `26.08.00a990`, H100 80GB, Python 3.12, CUDA 12.9, and `RLIMIT_NOFILE=131072`:

```
start: 5 FDs
after append 0: 126,038 FDs
append 1: RuntimeError: Unable to open file ... Too many open files
```

The same workload without `max_file_size` stays at about 2,038 FDs and completes. The result reproduced on node-local ext3 and WekaFS.

## Source indication

The current implementation appends a new `ParquetWriter` at [python/cudf/cudf/io/parquet.py#L2265-L2277](https://github.com/rapidsai/cudf/blob/main/python/cudf/cudf/io/parquet.py#L2265-L2277), while [close()](https://github.com/rapidsai/cudf/blob/main/python/cudf/cudf/io/parquet.py#L2279-L2289) closes every writer only at teardown.

## Expected behavior

Retired chunked writers should be closed as soon as their paths are rotated out, keeping descriptor usage bounded by the active partitions. `close()` should also release all remaining descriptors.

## Environment

- cuDF `26.08.00a990` nightly; not yet checked on a released build
- H100 80GB, CUDA 12.9, Python 3.12
- `RLIMIT_NOFILE` soft=hard=131,072

Contributor guide

Open the contributing guide

Research direction

Start in python/cudf/cudf/io/parquet.py around lines 2265-2289 and run the provided multi-partition reproducer to observe descriptor growth across appends. Trace how rotated writers are stored and how close() handles them. Done means rotated descriptors are released promptly, remaining writers are released by close(), and the workload completes without exhausting file descriptors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.