apache / apache/arrow-rs

Hoist resolved per-column writer properties across row groups

Open
#10,896 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge?**

#10880 added `WriterProperties::resolve_column_properties`, which collapses the
repeated `HashMap` lookups for a leaf column into a
single lookup that returns all resolved settings at once.

That resolution currently happens in `GenericColumnWriter::new`
(`parquet/src/column/writer/mod.rs`). Column writers are recreated for every row
group, so the cost is once per leaf column *per row group* rather than once per
leaf column per file. For a wide schema written as many row groups — e.g. 10,000
columns × 100 row groups — that is still a million hash lookups over the writer's
lifetime, even though every one of them returns the same answer.

**Describe the solution you'd like**

Resolve each leaf column's properties once when the file writer is constructed and
reuse the result for every row group. Concretely: cache a
`Vec` (indexed by leaf column) on
`SerializedFileWriter` / `ArrowWriter`, and thread the entry into
`get_column_writer` / `GenericColumnWriter::new` instead of having each writer look
it up from `WriterProperties`. Resolution then becomes once per leaf column per
file.

**Describe alternatives you've considered**

- Memoizing inside `WriterProperties` behind a lock — adds synchronization to a
type that is currently shared freely via `WriterPropertiesPtr`, and still costs a
lookup per row group.
- Leaving it as-is. The per-row-group cost is already far below the pre-#10880
behaviour, so this is an incremental win rather than a fix; it matters most for
wide-schema, many-row-group workloads.

**Additional context**

Follow-up from https://github.com/apache/arrow-rs/pull/10880 (see
https://github.com/apache/arrow-rs/pull/10880#issuecomment-5443041691).
`parquet/benches/writer_overhead.rs` already covers wide schemas; measuring this
would want a variant that writes multiple row groups, since the existing cases
write a single row group each.

Contributor guide

Open the contributing guide

Research direction

Start in parquet/src/column/writer/mod.rs, then trace SerializedFileWriter, ArrowWriter, get_column_writer, and GenericColumnWriter::new. Follow how leaf columns and WriterProperties are currently resolved, and inspect parquet/benches/writer_overhead.rs for the existing wide-schema benchmark. Done means resolved properties are reused across row groups and a multi-row-group benchmark measures the change.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.