apache / apache/beam

Consider bundling multiple ValidatesRunner tests into one pipeline

Open
#18,477 0 comments 0 reactions 0 assignees View on GitHub
improvement P3 tests
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

Currently ValidatesRunner test suites run 1 pipeline per unit test. That's a lot of small pipelines, and consumes a lot of resources especially in case of a pretty heavyweight runner like Dataflow, so tests take a long time and can't be run in parallel due to quota issues, etc.

[~jasonkuster] says he and [~davor] discussed that we could execute multiple unit tests in a single TestPipeline.

This JIRA is to track that idea.

To further develop it: in case of Java, we could create a custom JUnit Runner http://junit.org/junit4/javadoc/4.12/org/junit/runner/Runner.html that would apply all the transforms and PAsserts in unit tests to a single instance of TestPipeline (per class, rather than per method), and run the whole thing at the end. PAssert captures the source location of its application, so we could still report which particular test failed.

This obviously has fewer isolation between unit test methods, cause they effectively run in parallel instead of in sequence, so things like per-method setup and teardown will no longer be applicable. There'll probably be other issues.

Anyway, this seems doable and high-impact.

Imported from Jira [BEAM-2506](https://issues.apache.org/jira/browse/BEAM-2506). Original Jira may contain additional context.
Reported by: jkff.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the ValidatesRunner test suites and the JUnit Runner entry point mentioned in the issue. Investigate how multiple unit tests could share one TestPipeline per class while preserving PAssert failure locations, and assess the impact on per-method setup, teardown, and test isolation. Done means a validated design or implementation with those trade-offs addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.