apache / apache/airflow

Temporary disabling featuers in "canary" run due to problems with self-hosted runners

Open
#40,925 0 comments 1 reaction 0 assignees View on GitHub
area:self-hosted-runners kind:meta
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Body

Our self hosted runners are old (not updated since Sep 2023 - because the workflows in https://github.com/ashb/runner to update them have not been running. We temporarily switched the canary runs to run using public runners in:

* https://github.com/apache/airflow/pull/40913 followed up by
* https://github.com/apache/airflow/pull/40921

This has some limitations:

* [ ] the canary runs are running full tests and they take more than 1h 20 minutes to run (self-hosted were ~3/4 times faster
* [ ] they do not build and refresh cache for ARM images (work in progress in #41049 )
* [x] they run 2x a day currently as scheduled runs and can be manually triggered from https://github.com/apache/airflow/actions/workflows/ci.yml as needed (work in progress to increased to 4xDay by #41056 and it should be enough).
* [ ] they do not publish documentation because there is not enough disk space on public runners - PR https://github.com/apache/airflow/pull/41057
* [ ] we cannot release the multi-platform images via Github workflow (it requires public runner + ARM instance started by it)

This could be fixed in a varioous ways:

* when @ashb is back from holidays we can restore the old runners approach
* we can continue and complete work that @hussein-awala has done on using GitHub Kubernetes Controller to run ephemeral runner on our infrastructure
* some of the above (ARM) can be fixed by employing self-hosted ASF runners likely https://cwiki.apache.org/confluence/display/INFRA/ASF+Infra+provided+self-hosted+runners

### Committer

- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the ci.yml workflow and the referenced PRs #40913, #40921, #41049, #41056, and #41057 to understand the temporary public-runner setup and current work. Decide which runner approach should be pursued, then verify that canary runs can refresh ARM caches, publish documentation, and release multi-platform images without the listed limitations.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, kubernetes
Domain
ci-cd, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.