apache / apache/datafusion-comet

Native Parquet write of a zero-partition RDD produces no output file

Open
#5,303 0 comments 0 reactions 0 assignees View on GitHub
area:writer bug priority:medium
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Found while reviewing #5293 against the Iceberg split-writer work in #4658.

### Describe the bug

Spark's `WriteFilesExec.doExecuteWrite` guards against a child RDD with zero partitions (SPARK-23271):

```scala
val rddWithNonEmptyPartitions = if (rdd.partitions.length == 0) {
session.sparkContext.parallelize(Array.empty[InternalRow], 1)
} else {
rdd
}
```

Without it no write task runs at all, so the output directory ends up with no data file — not even the schema-only file Spark produces. `IcebergWriteExec` added in #4658 carries the same guard.

`CometWriteFilesExec.doExecuteWrite` has no equivalent: it calls `child.executeColumnar()` and goes straight to `mapPartitionsInternal`. With zero partitions, `FileFormatWriter.writeAndCommit` collects zero `WriteTaskResult`s and commits an empty job. The result is a `_SUCCESS` marker over an empty directory, where Spark writes one file carrying the schema.

Note this is not a regression from #5293 — the `CometNativeWriteExec` it replaced had the same hole.

### Steps to reproduce

Any write whose input plan yields a zero-partition RDD (for example a scan over an empty relation), with `spark.comet.parquet.write.enabled=true`.

### Expected behavior

Match Spark: run one write task so the output carries the schema.

Harder here than in the Iceberg case because the dummy RDD must be columnar — `parallelize(Array.empty[InternalRow], 1)` is row-based, and `CometWriteFilesExec` feeds batches into an Arrow stream. Options are a single-partition empty `ColumnarBatch` RDD, or bypassing the native writer for this case and writing the schema-only file on the JVM.

Contributor guide

Open the contributing guide

Research direction

Start in CometWriteFilesExec.doExecuteWrite and reproduce a zero-partition input with spark.comet.parquet.write.enabled=true. Compare its handling with Spark's WriteFilesExec and IcebergWriteExec, then determine how the columnar empty input should be handled. Done means the write produces the schema-only data file rather than only _SUCCESS.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.