apache / apache/datafusion-comet
Native Parquet write of a zero-partition RDD produces no output file
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Found while reviewing #5293 against the Iceberg split-writer work in #4658.
### Describe the bug
Spark's `WriteFilesExec.doExecuteWrite` guards against a child RDD with zero partitions (SPARK-23271):
```scala
val rddWithNonEmptyPartitions = if (rdd.partitions.length == 0) {
session.sparkContext.parallelize(Array.empty[InternalRow], 1)
} else {
rdd
}
```
Without it no write task runs at all, so the output directory ends up with no data file — not even the schema-only file Spark produces. `IcebergWriteExec` added in #4658 carries the same guard.
`CometWriteFilesExec.doExecuteWrite` has no equivalent: it calls `child.executeColumnar()` and goes straight to `mapPartitionsInternal`. With zero partitions, `FileFormatWriter.writeAndCommit` collects zero `WriteTaskResult`s and commits an empty job. The result is a `_SUCCESS` marker over an empty directory, where Spark writes one file carrying the schema.
Note this is not a regression from #5293 — the `CometNativeWriteExec` it replaced had the same hole.
### Steps to reproduce
Any write whose input plan yields a zero-partition RDD (for example a scan over an empty relation), with `spark.comet.parquet.write.enabled=true`.
### Expected behavior
Match Spark: run one write task so the output carries the schema.
Harder here than in the Iceberg case because the dummy RDD must be columnar — `parallelize(Array.empty[InternalRow], 1)` is row-based, and `CometWriteFilesExec` feeds batches into an Arrow stream. Options are a single-partition empty `ColumnarBatch` RDD, or bypassing the native writer for this case and writing the schema-only file on the JVM.
Contributor guide
Research direction
Start in CometWriteFilesExec.doExecuteWrite and reproduce a zero-partition input with spark.comet.parquet.write.enabled=true. Compare its handling with Spark's WriteFilesExec and IcebergWriteExec, then determine how the columnar empty input should be handled. Done means the write produces the schema-only data file rather than only _SUCCESS.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100