apache / apache/pinot

[Multi-stage] Stage stats are very misleading

Closed
#12,388 5 comments 1 reaction 1 assignee Claimed by @KKcorps View on GitHub
multi-stage observability stale
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently the stage stats reported in the query response is the aggregated stats over all the operators, which can be quite confusing.
When tracing is disabled, we collect stats for leaf and send operator, and reporting doubled `numDocs` and `numBlocks` than actually sent by the leaf stage.
When tracing is enabled, we collect stats for every operator, and aggregate `numDocs` and `numBlocks` for all the operators.

Instead, we might want to just record the stats for blocks sent out from each stage. We should also revisit other stats, and see if it makes sense to aggregate them.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.