facebookresearch / facebookresearch/fairo
Better surface parser failures
- Dominant language
- Jupyter Notebook
- Stars
- 929
- Forks
- 123
- PR merge metrics
- No merged PRs in 30d
Description
## Type of Issue
Select the type of issue:
- [ ] Bug report (to report a bug)
- [ ] Feature request (to request an additional feature)
- [x] Tracker (I am just using this as a tracker)
- [ ] Refactor request
- [ ] Documentation Ask
## Description
From conversation with @kavyasrinet: let's figure out a better way to surface and track parser failures.
Previous proposed solution was to have engineers directly write to ground truth using Autocomplete. Problem is that our grammar changes quickly, so the onboarding cost is not sustainable. It is also not sustainable to have a few people routinely add commands from issues to ground truth.
Another idea is to decide on a set of important commands. The problem with our lists of "high pri" commands now is that for everything deemed high pri, we already have it in ground truth. A more scalable approach might be to
1) Consider space of possible commands, instead of checking for exact matches. eg. "follow the X" might be essential for anything in the world that is X.
2) Do not add every failed command from an issue to ground truth, only if it is within the space of commands we really need in ground truth.
3) Alternatively, we don't add commands to ground truth from model issue trackers, and instead add them to model unit test. Test is not blocking but provides signal into how current model is doing.
## Checklist
TBD, will add subtasks once this is clear
- [ ] Come up with a scalable approach for handling model failures.
- [ ] Implement solution
Contributor guide
Assessment
This issue has not been assessed yet.