facebookresearch / facebookresearch/fairo
Current local test framework doesn't check for local artifacts diverging from remote
- Dominant language
- Jupyter Notebook
- Stars
- 929
- Forks
- 123
- PR merge metrics
- No merged PRs in 30d
Description
## Type of Issue
Select the type of issue:
- [ ] Bug report (to report a bug)
- [x] Feature request (to request an additional feature)
- [ ] Tracker (I am just using this as a tracker)
- [ ] Refactor request
- [ ] Documentation Ask
## Description
Right now the basic test framework (at least for `craftassist`) doesn't trigger a warning when the local dataset/ model are out of sync with remote. So you could be working on an old model / datasets and running the tests against those without even realizing.
Right now the checksum comparison and forced download only happens when we run for example : `craftassist_agent.py`
The proposal is : the tests should mimic what the agents do - so at the least we should check if checksum change exists at all - if yes warn and print an error message that please know that the local checksum differs from what's in `default_checksums/`
Because the problem is we are tracking the files in `default_checksums/` but the comparison only gets triggered when we run the agent. We need to at least notify the user of this during the tests. Because right now we can track what master supports but the test framework doesn't trigger a warning if my local is off and will silently try running all the tests against my local.
## Steps to reproduce
```
cd droidlet
cd craftassist
./test.sh
```
## Links to any relevant pastes or documents
For example my datasets were up-to-date but I spent a lot of time getting stuck on this error (and going through different pytorch versions) : https://www.internalfb.com/phabricator/paste/view/P400156878 without realizing my model was out of sync.
Contributor guide
Assessment
This issue has not been assessed yet.