apache / apache/beam

[Task]: Build RC and non-RC wheels in parallel.

Open
#27,368 2 comments 0 reactions 0 assignees View on GitHub
new feature P3 python release stale
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

code path: https://github.com/apache/beam/blob/master/.github/workflows/build_wheels.yml

There is exactly same steps "Build Wheel" and "Build RC wheels", both run the same command and produce same wheels with different naming:

```
8 wheels produced in 52 minutes:
apache_beam-2.49.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl 13,728 kB
apache_beam-2.49.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 14,286 kB
apache_beam-2.49.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl 14,585 kB
apache_beam-2.49.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 15,234 kB
apache_beam-2.49.0-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl 14,369 kB
apache_beam-2.49.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 14,945 kB
apache_beam-2.49.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl 13,999 kB
apache_beam-2.49.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 14,564 kB

8 wheels produced in 51 minutes:
apache_beam-2.49.0rc1-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl 13,728 kB
apache_beam-2.49.0rc1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 14,286 kB
apache_beam-2.49.0rc1-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl 14,585 kB
apache_beam-2.49.0rc1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 15,234 kB
apache_beam-2.49.0rc1-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl 14,369 kB
apache_beam-2.49.0rc1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 14,945 kB
apache_beam-2.49.0rc1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl 13,999 kB
apache_beam-2.49.0rc1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 14,564 kB
```

Example run:

https://github.com/apache/beam/actions/runs/5467460207/jobs/9953842597

This becomes extremely time consuming as linux-aarch64 wheels costs 2.5 h to run, then the workflow costs >5 h to run.

### Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

### Issue Components

- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Research direction

Read .github/workflows/build_wheels.yml and compare the "Build Wheel" and "Build RC wheels" steps, then inspect the linked example run to understand their dependencies. Done means RC and non-RC wheel builds run in parallel, both produce the expected wheels, and the workflow completes without breaking its existing outputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
build-system, ci-cd
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.