ray-project / ray-project/ray

[Data] Add a target output file size knob for write ops

Open
#65,327 2 comments 0 reactions 1 assignee View on GitHub

@laysfire is already working on this.

Since Aug 12, 2026.

data enhancement
Dominant language
Python
Stars
43.9k
Forks
8.1k
PR merge metrics
PR metrics pending

Description

Description

Description

Ray Data write ops (write_parquet, etc.) currently expose only min_rows_per_file and max_rows_per_file to control output file sizing. Because these are row-count based, output file sizes can vary dramatically when row byte-width varies . This makes it hard to produce evenly-sized output files.

Proposal

Add a size-based knob, e.g. target_file_size / target_block_size (bytes), that Ray Data uses to decide when to roll over to a new output file, estimating rows-per-file from observed average row size. This would sit alongside the existing row-based knobs (which can act as hard bounds).

So api will look something like ds.write_parquet(path, target_file_size="128MiB")

Use case

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.