asteroid-team / asteroid-team/asteroid
Automated tests for egs
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 450
- PR merge metrics
- No merged PRs in 30d
Description
## 🚀 Feature
Add automated tests for egs
### Motivation
Currently these are entirely untested, which means any changes done to the `run.sh` scripts etc. must be tested manually. When making changes, testing all the egs is a heavy burden on the developer; some of them even have commercial licenses that not everyone may have access to.
It also means that refactoring egs code takes much more time than could be.
### What you'd like
Add a "CI" mode to each eg:
- Run mini dataset preprocessing (maybe with autogenerated garbage source files so you do not need the actual dataset?)
- Run a mini epoch with a few batches
- Run eval
Automatically run this in CI for each eg.
Feedback welcome! It know it's a lot of work but we could easily split it into small steps.
Contributor guide
Research direction
Start by inventorying the egs and their run.sh scripts, then identify how each currently handles preprocessing, training, and evaluation. Define a minimal CI path for each eg that uses a small or generated dataset, runs a few batches, and performs evaluation; done means these checks run automatically in CI for every eg.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch, shell
- Domain
- ci-cd, machine-learning, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100