[Data] Add a target output file size knob for write ops
@laysfire is already working on this.
Since Aug 12, 2026.
- Dominant language
- Python
- Stars
- 43.9k
- Forks
- 8.1k
- PR merge metrics
- PR metrics pending
Description
Description
Description
Ray Data write ops (write_parquet, etc.) currently expose only min_rows_per_file and max_rows_per_file to control output file sizing. Because these are row-count based, output file sizes can vary dramatically when row byte-width varies . This makes it hard to produce evenly-sized output files.
Proposal
Add a size-based knob, e.g. target_file_size / target_block_size (bytes), that Ray Data uses to decide when to roll over to a new output file, estimating rows-per-file from observed average row size. This would sit alongside the existing row-based knobs (which can act as hard bounds).
So api will look something like ds.write_parquet(path, target_file_size="128MiB")
Use case
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.