apache / apache/datafusion-comet
Decide keep-or-lift for each remaining native Iceberg write eligibility restriction
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### What is the problem the feature request solves?
The native Iceberg write eligibility gate (`CometIcebergNativeWrite`, documented in `iceberg-writes.md`) is an allowlist, and each restriction it enforces was added conservatively in #5298 so the writer could land incrementally. None of the restrictions has an issue saying whether it is permanent or what would lift it, so there is no way to tell how much of the fallback surface is intended.
Current restrictions, each of which makes a write fall back to iceberg-java:
- [ ] format version 3 or newer (`MinUnsupportedFormatVersion`)
- [ ] any `encryption.*` property
- [ ] `write.parquet.bloom-filter-max-bytes` and per-column bloom filter enablement
- [ ] `write.parquet.page-version=v2`
- [ ] `write.parquet.shred-variants=true` (also depends on #5433 / the Variant epic #5438)
- [ ] any unvetted `write.parquet.*` key, and any `parquet.*` table property (including `parquet.enable.dictionary`)
- [ ] any `parquet.*` key in the session Hadoop configuration (other than `parquet.hadoop.vectored.io.enabled`)
- [ ] non-default `write.parquet.row-group-check-min-record-count` / `max-record-count`
- [ ] `write.object-storage.enabled=true` and `write.location-provider.impl`
- [ ] `io-impl` set, or a `FileIO` outside the recognized class hierarchy
- [ ] data location schemes other than `file`, `memory`, `s3`, `s3a`, `gs` (`oss` falls back because `oss.*` properties are not forwarded; see #5637 for `gs`)
### Describe the potential solution
For each restriction, record one of three decisions in this issue and in `iceberg-writes.md`:
- **Lift**: the native path already reproduces iceberg-java's outcome, or can with a bounded change; file a follow-up and link it.
- **Keep, pending upstream**: needs an iceberg-rust or parquet-rs feature; link the upstream issue.
- **Keep, permanent**: intentionally out of scope (for example encryption); say why.
The ordering should follow how often the setting appears in real tables. Object-storage layout (`write.object-storage.enabled`) and bloom filters are the two most likely to be hit by production tables and should be decided first.
### Additional context
Part of the native Iceberg writes epic, #5649. Related: #5298 (introduced the allowlist), #5361 (native writer).
Contributor guide
Research direction
Start with CometIcebergNativeWrite and iceberg-writes.md, then review #5298, #5361, #5433, #5438, and #5637 for the restrictions and related context. Compare the native and iceberg-java outcomes, prioritizing object-storage layout and bloom filters. Done means every listed restriction has a Lift, Keep pending upstream, or Keep permanent decision documented with the relevant follow-up or upstream issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala
- Domain
- backend, documentation
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100