apache / apache/airflow

Setup Github Runners via Kubernetes Controller on our AWS instance

Open
#41,388 0 comments 0 reactions 0 assignees View on GitHub
area:self-hosted-runners kind:meta
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Body

We should set-up the https://github.com/actions/actions-runner-controller on Airflow AWS instance - so that it can speed up our builds and allow for ARM builds to be prepared.

Currently we only utillse public runners and (for docs publishing only) - self-hosted runners of the ASF infrastructure (but there are far to small number of those to rely on anything else.

We need to implement the runners - especially that we have a lot of credits from AWS now.

The work is mostly done by @hussein-awala and he will be able to share the status with anyone who picks it up.

### Committer

- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Research direction

Start by contacting @hussein-awala for the current status, then review the actions-runner-controller setup on the Airflow AWS instance and the existing public and ASF self-hosted runner arrangements. Done means the controller and runners are implemented sufficiently to support builds, including ARM builds, using the available AWS capacity.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, github-actions, kubernetes
Domain
ci-cd, cloud, devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.