asteroid-team / asteroid-team/asteroid

Automated tests for egs

Open
#183 2 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Python
Stars
2.6k
Forks
450
PR merge metrics
No merged PRs in 30d

Description

## 🚀 Feature

Add automated tests for egs

### Motivation

Currently these are entirely untested, which means any changes done to the `run.sh` scripts etc. must be tested manually. When making changes, testing all the egs is a heavy burden on the developer; some of them even have commercial licenses that not everyone may have access to.

It also means that refactoring egs code takes much more time than could be.

### What you'd like

Add a "CI" mode to each eg:

- Run mini dataset preprocessing (maybe with autogenerated garbage source files so you do not need the actual dataset?)
- Run a mini epoch with a few batches
- Run eval

Automatically run this in CI for each eg.

Feedback welcome! It know it's a lot of work but we could easily split it into small steps.

Contributor guide

Open the contributing guide

Research direction

Start by inventorying the egs and their run.sh scripts, then identify how each currently handles preprocessing, training, and evaluation. Define a minimal CI path for each eg that uses a small or generated dataset, runs a few batches, and performs evaluation; done means these checks run automatically in CI for every eg.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch, shell
Domain
ci-cd, machine-learning, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.