Hoist resolved per-column writer properties across row groups
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge?**
#10880 added `WriterProperties::resolve_column_properties`, which collapses the
repeated `HashMap` lookups for a leaf column into a
single lookup that returns all resolved settings at once.
That resolution currently happens in `GenericColumnWriter::new`
(`parquet/src/column/writer/mod.rs`). Column writers are recreated for every row
group, so the cost is once per leaf column *per row group* rather than once per
leaf column per file. For a wide schema written as many row groups — e.g. 10,000
columns × 100 row groups — that is still a million hash lookups over the writer's
lifetime, even though every one of them returns the same answer.
**Describe the solution you'd like**
Resolve each leaf column's properties once when the file writer is constructed and
reuse the result for every row group. Concretely: cache a
`Vec` (indexed by leaf column) on
`SerializedFileWriter` / `ArrowWriter`, and thread the entry into
`get_column_writer` / `GenericColumnWriter::new` instead of having each writer look
it up from `WriterProperties`. Resolution then becomes once per leaf column per
file.
**Describe alternatives you've considered**
- Memoizing inside `WriterProperties` behind a lock — adds synchronization to a
type that is currently shared freely via `WriterPropertiesPtr`, and still costs a
lookup per row group.
- Leaving it as-is. The per-row-group cost is already far below the pre-#10880
behaviour, so this is an incremental win rather than a fix; it matters most for
wide-schema, many-row-group workloads.
**Additional context**
Follow-up from https://github.com/apache/arrow-rs/pull/10880 (see
https://github.com/apache/arrow-rs/pull/10880#issuecomment-5443041691).
`parquet/benches/writer_overhead.rs` already covers wide schemas; measuring this
would want a variant that writes multiple row groups, since the existing cases
write a single row group each.
Contributor guide
Research direction
Start in parquet/src/column/writer/mod.rs, then trace SerializedFileWriter, ArrowWriter, get_column_writer, and GenericColumnWriter::new. Follow how leaf columns and WriterProperties are currently resolved, and inspect parquet/benches/writer_overhead.rs for the existing wide-schema benchmark. Done means resolved properties are reused across row groups and a multi-row-group benchmark measures the change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100