[VL] Add details to the "time of input iterator" metric
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
For the "time of input iterator" metric in the `InputIteratorTransformer`, the representation of this time depends on its previous operator.
Below are the 3 different cases that I observe:
1. **When the previous operator is shuffle:**
The time is primarily the total shuffle read time, including fetch wait time and native reader processing time (such as decompression and deserialization).
2. **When the previous operator is broadcast:**
The time is nearly zero because the broadcast is already executed before the pipeline starts.
3. **For other cases (e.g., ColumnarUnion or fallback operators within the same Spark stage as the previous Velox pipelines):**
Since wallTimeNanos in Velox is measured by the driver’s getOutput, the time of the previous pipelines is included in the getOutput call from the ValueStreamNode. In this case, the time of input iterators represents the total time counted from the beginning of the current stage.
The discrepancy in behavior across different cases for this operator is not documented and may cause confusion for users. It would be better to document this and highlight it in the metrics description.
### Gluten version
None
Contributor guide
Research direction
Locate the InputIteratorTransformer metric description and read how wallTimeNanos and ValueStreamNode are used. Update the documentation to distinguish shuffle, broadcast, and other preceding-operator cases, including what the input-iterator time represents in each case; the metric description should make these differences explicit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- documentation, observability
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100