apache / apache/datafusion-comet
Decide and document the delta-spark version pin for the Delta contrib
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Follow-up from review of #4952 ([tracking comment](https://github.com/apache/datafusion-comet/pull/4366#issuecomment-5320446796), item 3).
### What is pinned today
The `contrib-delta` Maven profile pins a `delta-spark` version per Spark profile, in `spark/pom.xml`:
| Spark profile | `delta.version` |
|---|---|
| spark-3.5 | 3.3.2 |
| spark-4.0 | 4.0.0 |
| spark-4.1 | 4.1.0 |
with a top-level default of `4.1.0` in the root `pom.xml`. `dev/verify-contrib-delta-gate.sh`
asserts the mapping holds (spark-4.1 -> delta-spark 4.1.x, spark-3.5 -> 3.x, spark-4.0 -> 4.0.x)
and fails the build gate otherwise.
### The gap
The pin is a real compatibility decision — the contrib reflects into delta-spark internals
(`TahoeLogFileIndex`, `PreparedDeltaFileIndex`, `TahoeBatchFileIndex`, DV plumbing), so the
supported range is narrower than "any delta-spark". But that decision is currently only implicit
in the poms plus an assertion in the gate script. There is no issue recording:
- **which** delta-spark versions the contrib claims to support, and why those;
- **what a user should expect** if they run against a different delta-spark than the pinned one;
- **who updates the pin** when a new delta-spark ships, and what has to be re-verified (the
reflection surface is the risky part, not the wire format);
- whether the per-Spark mapping should ever become a **range** rather than an exact pin.
### Ask
Settle the policy and write it down — in the contrib user guide and/or the profile comments — and
keep the gate script assertion as the enforcement. This is a documentation/decision issue rather
than a code change; the current pins work.
---
🤖 Filed with [Claude Code](https://claude.com/claude-code).
Contributor guide
Research direction
Start by reading the delta.version mappings in spark/pom.xml and root pom.xml, then run dev/verify-contrib-delta-gate.sh to understand the existing enforcement. Review the reflection surface named in the issue and the contrib user guide before documenting the supported pins, behavior for other versions, update ownership, and re-verification expectations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- build-system, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100