apache / apache/beam

[Task]: Reduce running time of beam_PostCommit_Java_VR_Dataflow_V2_Streaming

Open
#26,479 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage dataflow java P3 task tests
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What needs to happen?

Found when doing #26416.

The test was added back in #13251 by duplicating the batch VR test suite (and include some streaming only tags). Since then the timeout has been bumped twice 330 min -> 450 min -> 510 min. In #14747 it was commented that

> We should then revisit exactly what tests we need to run and how to speed them up and consume fewer resources.

For Dataflow runner v1, batch and streaming jobs use different workers, but for runner v2, they uses the same worker. Many VR tests are testing basic transform functionalities (e.g. create, view) not involving batch/streaming differences. And most tests PCollection are bounded and not Timestamped. Does it suffice to run batch VR for them?

If not, we should probably reduce the trigger interval for this test at least, by now it consistently costs 6.5-7 h to run: https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/ even exceeds the postcommit trigger interval (6 h).

### Issue Priority

Priority: 3 (nice-to-have improvement)

### Issue Components

- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [x] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.