Bind PartitionSpec to its schema
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
When writing logic for rewriting manifest files, it was found to be easy to hand partition specs around with the wrong schema reference: https://github.com/apache/iceberg-rust/pull/2367#discussion_r3314540014
This issue proposes to bind a `PartitionSpec` to a schema, such as by holding a reference to the Schema. Or, a better approach if there is one.
Ultimately, rough idea:
- Add schema to PartitionSpec
- Update SnapshotSummaryCollector's APIs: schema should be inferred from partition spec, not passed in
Contributor guide
Research direction
Start by tracing PartitionSpec and SnapshotSummaryCollector, then read the linked pull request discussion for the schema-mismatch context. Determine how PartitionSpec can retain or otherwise derive its schema, and identify the callers affected by removing the separately passed schema. Done means the API prevents mismatched schema references and SnapshotSummaryCollector obtains the schema from the partition spec.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100