[Multi-stage] Stage stats are very misleading
Closed
multi-stage
observability
stale
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Currently the stage stats reported in the query response is the aggregated stats over all the operators, which can be quite confusing.
When tracing is disabled, we collect stats for leaf and send operator, and reporting doubled `numDocs` and `numBlocks` than actually sent by the leaf stage.
When tracing is enabled, we collect stats for every operator, and aggregate `numDocs` and `numBlocks` for all the operators.
Instead, we might want to just record the stats for blocks sent out from each stage. We should also revisit other stats, and see if it makes sense to aggregate them.
Contributor guide
Assessment
This issue has not been assessed yet.