Add separate test job to run only time-bound tests in isolation
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Body
There are few tests that have some "timing" component - i.e they depend on how fast they are run. This might lead to flaky tests when the test are run in parallel to some other tests in various environments.
Ideally, we should avoid such tests, however sometimes it is necessary to have them. For example: https://github.com/apache/airflow/pull/32612 added test that checks how slow the CLI tests are. We likely won't avoid such tests.
However, it might be a good idea to not run those tests in the main test run when we run tests in parallell - we could easily add another job (only for sqlite, only running on sequential set of timing tests, for one Python version only). Those tests could be skipped in regular runs.
Such test job would have much more stable envirionment for such timing tests, it would also make it possible to dial-down some of the time limits we have there, because currently a number of those tests have much higher limits set to account for such "busy" environment.
That would require:
* adding a separate "timing_tests" folder and Timing test type and move such tests there
* exclude Timing test type from --parallel runs
* add a separate job to only run the Timing test type (sequentially)
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Contributor guide
Research direction
Start by locating the test classification used by --parallel and the CI configuration for test jobs. Define the timing_tests folder and Timing test type, exclude that type from parallel runs, and add a sequential job that runs only those tests; done means timing tests run separately and are omitted from regular parallel runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, testing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100