facebookresearch / facebookresearch/fairo

Current local test framework doesn't check for local artifacts diverging from remote

Open
#315 0 comments 0 reactions 2 assignees Claimed by @snyxan View on GitHub
P2
Dominant language
Jupyter Notebook
Stars
929
Forks
123
PR merge metrics
No merged PRs in 30d

Description

## Type of Issue

Select the type of issue:
- [ ] Bug report (to report a bug)
- [x] Feature request (to request an additional feature)
- [ ] Tracker (I am just using this as a tracker)
- [ ] Refactor request
- [ ] Documentation Ask

## Description

Right now the basic test framework (at least for `craftassist`) doesn't trigger a warning when the local dataset/ model are out of sync with remote. So you could be working on an old model / datasets and running the tests against those without even realizing.
Right now the checksum comparison and forced download only happens when we run for example : `craftassist_agent.py`

The proposal is : the tests should mimic what the agents do - so at the least we should check if checksum change exists at all - if yes warn and print an error message that please know that the local checksum differs from what's in `default_checksums/`
Because the problem is we are tracking the files in `default_checksums/` but the comparison only gets triggered when we run the agent. We need to at least notify the user of this during the tests. Because right now we can track what master supports but the test framework doesn't trigger a warning if my local is off and will silently try running all the tests against my local.

## Steps to reproduce

```
cd droidlet
cd craftassist
./test.sh
```

## Links to any relevant pastes or documents

For example my datasets were up-to-date but I spent a lot of time getting stuck on this error (and going through different pytorch versions) : https://www.internalfb.com/phabricator/paste/view/P400156878 without realizing my model was out of sync.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.