apache / apache/datafusion-comet
Add native support for CollectMetricsExec (df.observe) to preserve stage fusion
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
`CollectMetricsExec` (`df.observe(...)`) is not supported, and because it sits in the middle of a plan the fallback is disproportionately expensive: it splits the surrounding native stage in two, adding two extra columnar/row transitions on top of running the observation itself in Spark.
`df.observe` is the standard hook for data-quality instrumentation in ETL pipelines (row counts, null counts, constraint checks), so pipelines that add observability today pay for it twice.
### Describe the potential solution
A native pass-through operator that forwards batches unchanged while computing the observed aggregate expressions, reporting results back through the existing native metrics / accumulator plumbing so `QueryExecutionListener` observation events still fire with the same values. The aggregate expressions allowed in `observe` are restricted (no distinct, no non-deterministic), which keeps the surface small.
The main win is not the aggregation cost, it is restoring stage fusion so the operators above and below stay in one native stage.
### Additional context
Found while completing the operator coverage table in #5111.
Contributor guide
Assessment
This issue has not been assessed yet.