apache / apache/datafusion-comet

Native Parquet writes report row counts but not row contents to WriteTaskStatsTrackers

Open
#5,307 0 comments 0 reactions 0 assignees View on GitHub
area:writer enhancement
Dominant language
Scala
Stars
1.3k
Forks
375
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

`WriteTaskStatsTracker.newRow(filePath: String, row: InternalRow)` is a per-row callback. Comet's native write path has columnar batches, not `InternalRow`s, so `CometWriteFilesExec.recordRows` calls it `n` times with `InternalRow.empty` rather than materializing every row just to hand it straight back.

That is exactly right for `BasicWriteTaskStatsTracker`, the only implementation Spark ships, which ignores the row argument and just increments a counter (`BasicWriteStatsTracker.scala`). But a third-party tracker that inspects row contents — a Delta or custom-catalog stats collector, say — would silently compute its statistics over empty rows.

Today this logs a warning per non-`BasicWriteTaskStatsTracker` instance. A warning is the honest minimum, but it is not a guarantee.

### Describe the potential solution

The obstacle is that a plan-time guard is not possible: `WriteJobDescription.statsTrackers` only exists at execution time, by which point `getSupportLevel` has already accepted the write and there is no way to fall back.

Options:

1. Materialize rows only when a non-basic tracker is present — correct, and pays the row-conversion cost solely in the case that needs it.
2. Fail the write with a clear message instead of warning, so nobody gets wrong statistics silently.
3. Find a plan-time signal for the tracker set so the write can fall back to Spark gracefully.

Option 1 is the most useful; option 2 is a smaller step if the row-conversion path is not worth building yet.

### Additional context

Introduced by #5293, which moved native writes onto Spark's `WriteFilesExec` seam and therefore onto Spark's stats-tracker contract. The old path bypassed the trackers entirely and reported its own metrics, so this is a new obligation rather than a regression.

Contributor guide

Open the contributing guide

Research direction

Start at CometWriteFilesExec.recordRows and read BasicWriteStatsTracker.scala to compare the native callback behavior with Spark’s row-count-only tracker. Trace WriteJobDescription.statsTrackers at execution time and determine which behavior the implementation will guarantee for non-basic trackers; done means tracker row contents are handled safely without silently producing incorrect statistics.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.