apache / apache/datafusion-comet
Native Parquet writes report row counts but not row contents to WriteTaskStatsTrackers
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 375
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
`WriteTaskStatsTracker.newRow(filePath: String, row: InternalRow)` is a per-row callback. Comet's native write path has columnar batches, not `InternalRow`s, so `CometWriteFilesExec.recordRows` calls it `n` times with `InternalRow.empty` rather than materializing every row just to hand it straight back.
That is exactly right for `BasicWriteTaskStatsTracker`, the only implementation Spark ships, which ignores the row argument and just increments a counter (`BasicWriteStatsTracker.scala`). But a third-party tracker that inspects row contents — a Delta or custom-catalog stats collector, say — would silently compute its statistics over empty rows.
Today this logs a warning per non-`BasicWriteTaskStatsTracker` instance. A warning is the honest minimum, but it is not a guarantee.
### Describe the potential solution
The obstacle is that a plan-time guard is not possible: `WriteJobDescription.statsTrackers` only exists at execution time, by which point `getSupportLevel` has already accepted the write and there is no way to fall back.
Options:
1. Materialize rows only when a non-basic tracker is present — correct, and pays the row-conversion cost solely in the case that needs it.
2. Fail the write with a clear message instead of warning, so nobody gets wrong statistics silently.
3. Find a plan-time signal for the tracker set so the write can fall back to Spark gracefully.
Option 1 is the most useful; option 2 is a smaller step if the row-conversion path is not worth building yet.
### Additional context
Introduced by #5293, which moved native writes onto Spark's `WriteFilesExec` seam and therefore onto Spark's stats-tracker contract. The old path bypassed the trackers entirely and reported its own metrics, so this is a new obligation rather than a regression.
Contributor guide
Research direction
Start at CometWriteFilesExec.recordRows and read BasicWriteStatsTracker.scala to compare the native callback behavior with Spark’s row-count-only tracker. Trace WriteJobDescription.statsTrackers at execution time and determine which behavior the implementation will guarantee for non-basic trackers; done means tracker row contents are handled safely without silently producing incorrect statistics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100