apache / apache/gluten

[VL] Scan output metric is not accurate.

Open
#8,702 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

Sometimes, we will try to figure out the key contribution of the engine. The plan metrics is a good way to evaluate performance。
But we found that Scan metrics are inaccurate, leading to a series of misjudgments。

Q96 (1 TB scale) is one example.

![Image](https://github.com/user-attachments/assets/3169b517-5cef-4242-852f-1b3b314e6f8d)

Here: "number of output rows: 709, 525". This value is equal to the output of the third join, which is incredible.

Image

cc @zhouyuan @weiting-chen , Thanks.

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Reproduce Q96 at 1 TB scale and inspect the plan's scan metrics against the actual scan and join outputs shown in the report. Trace where the scan output-row value is produced; done means the scan metrics report the scan's own output accurately rather than a later join's output.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
backend, observability, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.