astropy / astropy/astroquery

More roboust remote testing

Open
#2,113 1 comment 0 reactions 0 assignees View on GitHub
enhancement infrastructure testing
Dominant language
Python
Stars
791
Forks
451
Avg merge
1d 3h
Merged PRs (30d)
4

Description

The example in #2112 unearthed the issue that our remote-testing protocol is not sufficient enough, we could have catch the bug if the CI matrix was running remote data tests for the old dependency combination, too.

There are a few options we could do, and I raise this issue with the infrastructure team to see what is the best way forward:

- set up a system to test on module level, and run remote data tests in PRs that are either triggered by:
- label
- automatically (but for the module only)

The main issue with remote testing is still that running all of them:
- takes a lot of time (1+ hour)
- stresses remote services even when changes is clearly unrelated
- there could be a lot of false positives, we don't want to have a constant failing CI just because one or two services have downtimes or issues

Contributor guide

Open the contributing guide

Research direction

Start with the example in issue #2112, then inspect the repository's CI matrix and remote-testing setup; no specific files or tests are named here. Done means the team has an implemented and documented approach for targeted remote data testing that covers relevant dependency combinations without imposing unnecessary runtime or service load.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, infrastructure, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.