apache / apache/datafusion-comet

Reconcile the two config namespaces for native writes

Open
#5,306 0 comments 0 reactions 0 assignees View on GitHub
area:writer enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### What is the problem the feature request solves?

Comet now has two experimental write features under two different config namespaces:

| Config | Added by |
| --- | --- |
| `spark.comet.write.iceberg.splitOperator.enabled` | #4658 |
| `spark.comet.parquet.write.enabled` | native Parquet writes |

One is `spark.comet.write..`, the other `spark.comet..write.`. Both are `CATEGORY_TESTING` and default to false, so nothing user-facing is broken yet — which makes this the cheap moment to pick one shape, before either graduates and the key has to be kept as a deprecated alias.

Worth settling alongside it: the operator opt-in for native Parquet writes is `spark.comet.operator.WriteFilesExec.allowIncompatible` (with `spark.comet.operator.DataWritingCommandExec.allowIncompatible` retained as an alternative), so a user enabling native Parquet writes currently sets two configs while the Iceberg feature needs one.

### Describe the potential solution

Pick one namespace and move the other with `withAlternative`, per the rename checklist in `docs/source/contributor-guide/config_conventions.md`. `spark.comet.write.*` reads better as writes grow more formats and features, but either is fine as long as it is one.

Contributor guide

Open the contributing guide

Research direction

Start with the rename checklist in docs/source/contributor-guide/config_conventions.md, then locate the definitions of the two native-write config keys named in the issue. Choose one namespace and use withAlternative for the other; check whether the Parquet operator opt-in config also needs reconciliation. Done means the keys follow one namespace convention and existing alternatives remain supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
developer-experience
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.