apache / apache/datafusion-comet

Add native support for CollectMetricsExec (df.observe) to preserve stage fusion

Open
#5,124 2 comments 0 reactions 1 assignee Claimed by @kazantsev-maksim View on GitHub
enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

`CollectMetricsExec` (`df.observe(...)`) is not supported, and because it sits in the middle of a plan the fallback is disproportionately expensive: it splits the surrounding native stage in two, adding two extra columnar/row transitions on top of running the observation itself in Spark.

`df.observe` is the standard hook for data-quality instrumentation in ETL pipelines (row counts, null counts, constraint checks), so pipelines that add observability today pay for it twice.

### Describe the potential solution

A native pass-through operator that forwards batches unchanged while computing the observed aggregate expressions, reporting results back through the existing native metrics / accumulator plumbing so `QueryExecutionListener` observation events still fire with the same values. The aggregate expressions allowed in `observe` are restricted (no distinct, no non-deterministic), which keeps the surface small.

The main win is not the aggregation cost, it is restoring stage fusion so the operators above and below stay in one native stage.

### Additional context

Found while completing the operator coverage table in #5111.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.