apache / apache/beam

Runner Comparison / Capability Matrix revamp

Open
#18,603 1 comment 0 reactions 0 assignees View on GitHub
improvement mentor P3 website
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

The goal for this project has changed: We now want to create a completely new Capability Matrix that is based on the ValidatesRunner tests that we run on the various Apache Beam runners.

We can use the test in ./test-infra/validates-runner/ to generate a JSON file that contains the capabilities supported by various runners and tested by each individual test.

\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--

 

Discussion: [https://lists.apache.org/thread.html/8aff7d70c254356f2dae3109fb605e0b60763602225a877d3dadf8b7@%3Cdev.beam.apache.org%3E](https://lists.apache.org/thread.html/8aff7d70c254356f2dae3109fb605e0b60763602225a877d3dadf8b7@%3Cdev.beam.apache.org%3E)

Summarizing that discussion, we have a lot of issues/wishes. Some can be addressed as one-off and some need a unified reorganization of the runner comparison.

Basic corrections:
- Remove rows that impossible to not support (ParDo)
- Remove rows where "support" doesn't really make sense (Composite transforms)
- Deduplicate rows are actually the same model feature (all non-merging windowing / all merging windowing)
- Clearly separate rows that represent optimizations (Combine)
- Correct rows in the wrong place (Timers are actually a "what...?" row)
- Separate or remove rows have not been designed ([Meta]Data driven triggers, retractions)
- Rename rows with names that appear no where else (Timestamp control, which is called a TimestampCombiner in Java)
- Switch to a more distinct color scheme for full/partial support (currently just solid/faded colors)
- Switch to something clearer than "~" for partial support, versus ✘ and ✓ for none and full.
- Correct Gearpump support for merging windows (see BEAM-2759)
- Correct Spark support for non-merging and merging windows (see BEAM-2499)

Minor rewrites:
- Lump all the basic stuff (ParDo, GroupByKey, Read, Window) into one row
- Make sections as users see them, like "ParDo" / "side Inputs" not "What?" / "side inputs"
- Add rows for non-model things, like portability framework support, metrics backends, etc

Bigger rewrites:
- Add versioning to the comparison, as in BEAM-166
- Find a way to fit in a plain English summary of runner's support in Beam. It should come first, as it is what new users need before getting to details.
- Find a way to describe production readiness of runners and/or testimonials of who is using it in production.
- Have a place to compare non-model differences between runners

Changes requiring engineering efforts:
- Gather and add quantitative runner metrics, perhaps Nexmark results for mid-level, smaller benchmarks for measuring aspects of specific features, and larger end-to-end benchmarks to get an idea how it might actually perform on a use case
- Tighter coupling of the matrix portion of the comparison with tags on ValidatesRunner tests

If you care to address some aspect of this, please reach out and/or just file a subtask and address it.

Imported from Jira [BEAM-2888](https://issues.apache.org/jira/browse/BEAM-2888). Original Jira may contain additional context.
Reported by: kenn.

Contributor guide

Open the contributing guide

Research direction

Start in ./test-infra/validates-runner/ to understand how runner capabilities are tested and generated. Read the linked mailing-list discussion and the referenced BEAM-2759, BEAM-2499, and BEAM-166 issues before narrowing the work. Done requires an agreed scope and a new capability matrix based on ValidatesRunner results, with documented support distinctions.

Written by the indexing model from the issue text.

Assessment

Domain
documentation, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.