[Task]: Build RC and non-RC wheels in parallel.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
code path: https://github.com/apache/beam/blob/master/.github/workflows/build_wheels.yml
There is exactly same steps "Build Wheel" and "Build RC wheels", both run the same command and produce same wheels with different naming:
```
8 wheels produced in 52 minutes:
apache_beam-2.49.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl 13,728 kB
apache_beam-2.49.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 14,286 kB
apache_beam-2.49.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl 14,585 kB
apache_beam-2.49.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 15,234 kB
apache_beam-2.49.0-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl 14,369 kB
apache_beam-2.49.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 14,945 kB
apache_beam-2.49.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl 13,999 kB
apache_beam-2.49.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 14,564 kB
8 wheels produced in 51 minutes:
apache_beam-2.49.0rc1-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl 13,728 kB
apache_beam-2.49.0rc1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 14,286 kB
apache_beam-2.49.0rc1-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl 14,585 kB
apache_beam-2.49.0rc1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 15,234 kB
apache_beam-2.49.0rc1-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl 14,369 kB
apache_beam-2.49.0rc1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 14,945 kB
apache_beam-2.49.0rc1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl 13,999 kB
apache_beam-2.49.0rc1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 14,564 kB
```
Example run:
https://github.com/apache/beam/actions/runs/5467460207/jobs/9953842597
This becomes extremely time consuming as linux-aarch64 wheels costs 2.5 h to run, then the workflow costs >5 h to run.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Contributor guide
Research direction
Read .github/workflows/build_wheels.yml and compare the "Build Wheel" and "Build RC wheels" steps, then inspect the linked example run to understand their dependencies. Done means RC and non-RC wheel builds run in parallel, both produce the expected wheels, and the workflow completes without breaking its existing outputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- build-system, ci-cd
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100