apache / apache/airflow

Add separate test job to run only time-bound tests in isolation

Open
#32,620 0 comments 0 reactions 0 assignees View on GitHub
kind:meta
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Body

There are few tests that have some "timing" component - i.e they depend on how fast they are run. This might lead to flaky tests when the test are run in parallel to some other tests in various environments.

Ideally, we should avoid such tests, however sometimes it is necessary to have them. For example: https://github.com/apache/airflow/pull/32612 added test that checks how slow the CLI tests are. We likely won't avoid such tests.

However, it might be a good idea to not run those tests in the main test run when we run tests in parallell - we could easily add another job (only for sqlite, only running on sequential set of timing tests, for one Python version only). Those tests could be skipped in regular runs.

Such test job would have much more stable envirionment for such timing tests, it would also make it possible to dial-down some of the time limits we have there, because currently a number of those tests have much higher limits set to account for such "busy" environment.

That would require:

* adding a separate "timing_tests" folder and Timing test type and move such tests there
* exclude Timing test type from --parallel runs
* add a separate job to only run the Timing test type (sequentially)

### Committer

- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Research direction

Start by locating the test classification used by --parallel and the CI configuration for test jobs. Define the timing_tests folder and Timing test type, exclude that type from parallel runs, and add a sequential job that runs only those tests; done means timing tests run separately and are omitted from regular parallel runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.