Improve Traceability of Pipeline translation
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Users often ask how they can reason about the pipeline translation. The Flink UI display a confusingly large graph without any trace of the original Beam pipeline:
WordCount:
!wordcount.png!
TFX:
!tfx.png!
Some aspects which make understanding these graphs hard:
* Users don't know how the Runner maps Beam to Flink concepts
* The UI is awfully slow / hangs when the pipeline is reasonable complex
* The operator names seem to use `transform.getUniqueName()` which doesn't generate readable name
* So called Chaining combines operators into a single operator which makes understanding which Beam concept belongs to which Flink concept even harder
Imported from Jira [BEAM-5859](https://issues.apache.org/jira/browse/BEAM-5859). Original Jira may contain additional context.
Reported by: mxm.
Contributor guide
Assessment
This issue has not been assessed yet.