asteroid-team / asteroid-team/asteroid

[CI] Add version compatibility tests for pretrained models

Open
#294 14 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
2.6k
Forks
450
PR merge metrics
No merged PRs in 30d

Description

## 🚀 Feature

Include version compat tests for pretrained models (models from asteroid 0.3.0 still work as expected in 0.3.3).

### Motivation

We can break things between versions and still pass CI, this is not ok.
For example #255 (solved by #258 )

### Questions & remarks
- Only testing against the previous released version seems enough to me.
- This means downloading all the models from Zenodo, it seems a bit heavy. It should probably be in a separate CI build.
- Probably need to install old version, load the models, separate one file and write to disk. Install current version. Make sure the model in the current version obtains the same output.
- Other possibility: have a tarball with reference/estimate file pairs for all models, on Zenodo. Download this tarball, test against model prediction in the current version. One _big_ downside of that is that we have to add reference/estimate for each model _by hand_ and update this tarball for each new model.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing CI configuration and the pretrained-model loading path. Compare the proposed previous-version installation approach with the Zenodo reference-tarball approach, including the cost of downloading all models. Done means CI verifies that models released with the previous version produce the same outputs under the current version.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
ci-cd, machine-learning, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.