apache / apache/datafusion-comet
Reconcile the two config namespaces for native writes
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### What is the problem the feature request solves?
Comet now has two experimental write features under two different config namespaces:
| Config | Added by |
| --- | --- |
| `spark.comet.write.iceberg.splitOperator.enabled` | #4658 |
| `spark.comet.parquet.write.enabled` | native Parquet writes |
One is `spark.comet.write..`, the other `spark.comet..write.`. Both are `CATEGORY_TESTING` and default to false, so nothing user-facing is broken yet — which makes this the cheap moment to pick one shape, before either graduates and the key has to be kept as a deprecated alias.
Worth settling alongside it: the operator opt-in for native Parquet writes is `spark.comet.operator.WriteFilesExec.allowIncompatible` (with `spark.comet.operator.DataWritingCommandExec.allowIncompatible` retained as an alternative), so a user enabling native Parquet writes currently sets two configs while the Iceberg feature needs one.
### Describe the potential solution
Pick one namespace and move the other with `withAlternative`, per the rename checklist in `docs/source/contributor-guide/config_conventions.md`. `spark.comet.write.*` reads better as writes grow more formats and features, but either is fine as long as it is one.
Contributor guide
Research direction
Start with the rename checklist in docs/source/contributor-guide/config_conventions.md, then locate the definitions of the two native-write config keys named in the issue. Choose one namespace and use withAlternative for the other; check whether the Parquet operator opt-in config also needs reconciliation. Done means the keys follow one namespace convention and existing alternatives remain supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- developer-experience
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100