More roboust remote testing
- Dominant language
- Python
- Stars
- 791
- Forks
- 451
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 4
Description
The example in #2112 unearthed the issue that our remote-testing protocol is not sufficient enough, we could have catch the bug if the CI matrix was running remote data tests for the old dependency combination, too.
There are a few options we could do, and I raise this issue with the infrastructure team to see what is the best way forward:
- set up a system to test on module level, and run remote data tests in PRs that are either triggered by:
- label
- automatically (but for the module only)
The main issue with remote testing is still that running all of them:
- takes a lot of time (1+ hour)
- stresses remote services even when changes is clearly unrelated
- there could be a lot of false positives, we don't want to have a constant failing CI just because one or two services have downtimes or issues
Contributor guide
Research direction
Start with the example in issue #2112, then inspect the repository's CI matrix and remote-testing setup; no specific files or tests are named here. Done means the team has an implemented and documented approach for targeted remote data testing that covers relevant dependency combinations without imposing unnecessary runtime or service load.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, infrastructure, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100