apache / apache/gluten

[VL] Add details to the "time of input iterator" metric

Open
#10,618 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

For the "time of input iterator" metric in the `InputIteratorTransformer`, the representation of this time depends on its previous operator.

Below are the 3 different cases that I observe:

1. **When the previous operator is shuffle:**
The time is primarily the total shuffle read time, including fetch wait time and native reader processing time (such as decompression and deserialization).
2. **When the previous operator is broadcast:**
The time is nearly zero because the broadcast is already executed before the pipeline starts.
3. **For other cases (e.g., ColumnarUnion or fallback operators within the same Spark stage as the previous Velox pipelines):**
Since wallTimeNanos in Velox is measured by the driver’s getOutput, the time of the previous pipelines is included in the getOutput call from the ValueStreamNode. In this case, the time of input iterators represents the total time counted from the beginning of the current stage.

The discrepancy in behavior across different cases for this operator is not documented and may cause confusion for users. It would be better to document this and highlight it in the metrics description.

### Gluten version

None

Contributor guide

Open the contributing guide

Research direction

Locate the InputIteratorTransformer metric description and read how wallTimeNanos and ValueStreamNode are used. Update the documentation to distinguish shuffle, broadcast, and other preceding-operator cases, including what the input-iterator time represents in each case; the metric description should make these differences explicit.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
documentation, observability
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.