apache / apache/datafusion-comet
[EPIC] Production-quality native Iceberg writes
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Native Iceberg writes landed in three parts under #4322: the split writer/committer plan (#4658), the eligibility allowlist with fall-back reasons (#5298), and the iceberg-rust data-file writer with JVM-side manifest metrics (#5361). #4322 closed when #5361 merged, and no issue tracks what remains between "merged behind two off-by-default flags" and "production quality". The open follow-ups exist but are not linked to each other, and several gaps had no issue at all.
This epic is that plan. The goal is unchanged from #4322: an ETL job that scans, transforms, and writes Iceberg tables runs natively end to end, and falls back to iceberg-java wherever the native path cannot reproduce its outcome.
**Current state.** `spark.comet.write.iceberg.splitOperator.enabled` and `spark.comet.iceberg.write.enabled` both default to `false`. Nothing in CI runs with either flag on. Two correctness bugs are open on the native path. The user guide (`iceberg-writes.md`) still describes the writer as unimplemented in one section.
### Describe the potential solution
Phased, in priority order. Each item is a linked issue; check it here when it closes.
**Phase 1: correctness blockers**
- [ ] #5636 Bump the iceberg-rust pin past the partition-path escaping fix (a `#` in an S3 partition value commits a location iceberg-java resolves to a different key)
- [ ] #5637 Native writes drop Hadoop GCS configuration for `HadoopFileIO` + `gs://`; fail closed until the translation is complete
- [ ] #5641 Fix the stale "not yet wired in" wording and the divergence list in `iceberg-writes.md`
- [ ] #5719 `revertToSpark` erases the write node, because `CometIcebergWriteExec` and `CometNativeWriteExec` report their own child as `originalPlan` (opt-in: needs `spark.comet.exec.transitionRevert.enabled`)
**Phase 2: failure handling parity with iceberg-java**
- [ ] #5618 Clean up task-attempt data files on task failure
- [ ] #5277 Remove orphaned files when the commit fails
- [ ] #5646 Failure-injection tests that assert both the visible outcome and the storage state
**Phase 3: coverage so common workloads stay native**
- [ ] #5635 Iceberg system functions (`bucket`, `truncate`, `years`/`months`/`days`/`hours`) as native expressions, so the default hash-distribution write is native end to end (PR #5638)
- [ ] #5318 Native `MergeRowsExec`
- [ ] #5122 Row-level MERGE / UPDATE / DELETE plans
- [ ] #5643 A keep-or-lift decision for every remaining eligibility restriction, so the fallback surface is intentional
- [ ] #5306 Reconcile the two config namespaces for native writes
**Phase 4: enable by default**
- [ ] #5259 Fix the four CI failure buckets when the split plan is on by default
- [ ] #5644 Graduation criteria and the two-step default flip (split plan first, native writer one release later), including a CI job that runs the Iceberg suites with both flags on
**Phase 5: performance**
- [ ] #5647 A native Iceberg write benchmark in the repository (unpartitioned, default hash distribution, fanout, copy-on-write delete)
- [ ] #5648 Account the writer's buffers in Comet's memory pool, especially the fanout writer's per-partition buffers
- [ ] #3595 Fused sort plus write for the clustered path
**Ongoing**
- [ ] #5645 A pin-bump policy for iceberg-rust, so upstream fixes to file bytes, manifests, or partition layout reach Comet on a schedule
- [ ] #3756 Update the Iceberg feature matrix, which still lists the write path as not implemented
### Additional context
Done and merged: #4658 (split-operator plan), #5298 (eligibility detection), #5361 (native writer). Related umbrellas: #5121 (DataSource V2 writes in general), #1625 (native Parquet writes for the V1 path).
The accepted divergences between iceberg-rust and iceberg-java output are documented in `iceberg-writes.md`; changes to that list belong in the issues above rather than here.
Contributor guide
Research direction
Start with iceberg-writes.md and the two native-write flags, then use the linked Phase 1–5 issues to select a self-contained task. Review the relevant issue before changing code, and run the Iceberg suites or checks it names. Done means the selected follow-up is complete and its checkbox can be marked without taking on the whole production-readiness plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100