apache / apache/datafusion-comet

Adopt a pin-bump policy for the iceberg-rust dependency

Open
#5,645 0 comments 0 reactions 0 assignees View on GitHub
area:Iceberg enhancement requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### What is the problem the feature request solves?

`native/Cargo.toml` pins `iceberg` and `iceberg-storage-opendal` to a git revision (`3d84c81353b1b23b6e4ae8eea8f8a021cc6927a7`) rather than a release. The pin is what the native Iceberg scan and writer are tested against, but nothing decides when it moves, and #5636 is the first bug caused by it drifting behind an upstream fix (partition-path escaping, apache/iceberg-rust#2875). With the writer now producing files that iceberg-java reads back, a stale pin can turn an upstream bug fix into a Comet correctness bug.

### Describe the potential solution

Write down and follow a policy, for example:

- Bump the pin at least once per Comet release cycle, and immediately when an upstream fix affects file bytes, manifest metadata, or partition layout.
- Prefer tagged iceberg-rust releases over arbitrary revisions once a release contains everything Comet needs, so the version is visible in `Cargo.lock` and in the docs.
- Each bump runs the Iceberg suites on all four Spark/Iceberg profiles plus the manifest-metrics parity tests, since those are what catch behavior changes in the writer.
- Track "waiting on upstream" items (see the eligibility-restrictions issue) so a bump can also lift restrictions.
- Note the bump in the release notes, since users reading Comet-written tables with iceberg-java care which writer semantics they got.

### Additional context

Part of the native Iceberg writes epic, #5649. Related: #5636.

Contributor guide

Open the contributing guide

Research direction

Start with native/Cargo.toml and the existing Iceberg suites for all four Spark/Iceberg profiles, plus the manifest-metrics parity tests. Define the pin-bump cadence, upstream-fix handling, tagged-release preference, waiting-on-upstream tracking, and release-note requirement; done means the policy is written and its validation and release steps are explicit.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, spark
Domain
data-engineering, release, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.