apache / apache/datafusion-comet
Adopt a pin-bump policy for the iceberg-rust dependency
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### What is the problem the feature request solves?
`native/Cargo.toml` pins `iceberg` and `iceberg-storage-opendal` to a git revision (`3d84c81353b1b23b6e4ae8eea8f8a021cc6927a7`) rather than a release. The pin is what the native Iceberg scan and writer are tested against, but nothing decides when it moves, and #5636 is the first bug caused by it drifting behind an upstream fix (partition-path escaping, apache/iceberg-rust#2875). With the writer now producing files that iceberg-java reads back, a stale pin can turn an upstream bug fix into a Comet correctness bug.
### Describe the potential solution
Write down and follow a policy, for example:
- Bump the pin at least once per Comet release cycle, and immediately when an upstream fix affects file bytes, manifest metadata, or partition layout.
- Prefer tagged iceberg-rust releases over arbitrary revisions once a release contains everything Comet needs, so the version is visible in `Cargo.lock` and in the docs.
- Each bump runs the Iceberg suites on all four Spark/Iceberg profiles plus the manifest-metrics parity tests, since those are what catch behavior changes in the writer.
- Track "waiting on upstream" items (see the eligibility-restrictions issue) so a bump can also lift restrictions.
- Note the bump in the release notes, since users reading Comet-written tables with iceberg-java care which writer semantics they got.
### Additional context
Part of the native Iceberg writes epic, #5649. Related: #5636.
Contributor guide
Research direction
Start with native/Cargo.toml and the existing Iceberg suites for all four Spark/Iceberg profiles, plus the manifest-metrics parity tests. Define the pin-bump cadence, upstream-fix handling, tagged-release preference, waiting-on-upstream tracking, and release-note requirement; done means the policy is written and its validation and release steps are explicit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, spark
- Domain
- data-engineering, release, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100