apache / apache/beam

DirectRunner RunnableOnService tempLocation configuration insufficient

Open
#18,010 0 comments 0 reactions 0 assignees View on GitHub
bug direct P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

The way we specify temp storage locations for RunnableOnService tests is not sufficient, specifically for DirectRunner execution. Right now, RunnableOnService tests are run for DirectRunner and DataflowRunner, who set their temp locations differently:

* DirectRunner doesn't specify a temp location directly, but test classes will use a JUnit @Rule TemporaryDirectory. Individual tests set it as necessary for tempLocation, and set a fake gs:// path for individual GCP IO tests.
* DataflowRunner tests pass an actual GCS path as tempRoot, and TestDataflowRunner will initialize stagingLocation to this path.

This setup makes it difficult to write RunnableOnService tests which pass for both runners. We should separate temp location setup out of individual test classes so that RunnableOnService tests "just work" on any runner.

One solution would be to add logic inside TestPipeline#testingPipelineOptions:

- If \--tempRoot is specified, use it to set tempLocation and stagingLocation. Otherwise, use a JUnit TemporaryDirectory to set it
- If tempLocation is a GCS path, use it to set stagingLocation. Otherwise, use a fake gcs path (i.e. gs://foo)

Imported from Jira [BEAM-436](https://issues.apache.org/jira/browse/BEAM-436). Original Jira may contain additional context.
Reported by: swegner.

Contributor guide

Open the contributing guide

Research direction

Start with TestPipeline#testingPipelineOptions and compare the existing DirectRunner setup with TestDataflowRunner and RunnableOnService tests. Trace how --tempRoot, TemporaryDirectory, tempLocation, and stagingLocation are currently supplied. Done means RunnableOnService tests can use consistent temporary locations under both DirectRunner and DataflowRunner without per-test setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.