facebookresearch / facebookresearch/fairo

Better surface parser failures

Open
#305 1 comment 0 reactions 1 assignee Claimed by @Rebecca-Qian View on GitHub
hack-session-worthy NLU P2 tracker
Dominant language
Jupyter Notebook
Stars
929
Forks
123
PR merge metrics
No merged PRs in 30d

Description

## Type of Issue

Select the type of issue:
- [ ] Bug report (to report a bug)
- [ ] Feature request (to request an additional feature)
- [x] Tracker (I am just using this as a tracker)
- [ ] Refactor request
- [ ] Documentation Ask

## Description

From conversation with @kavyasrinet: let's figure out a better way to surface and track parser failures.

Previous proposed solution was to have engineers directly write to ground truth using Autocomplete. Problem is that our grammar changes quickly, so the onboarding cost is not sustainable. It is also not sustainable to have a few people routinely add commands from issues to ground truth.

Another idea is to decide on a set of important commands. The problem with our lists of "high pri" commands now is that for everything deemed high pri, we already have it in ground truth. A more scalable approach might be to
1) Consider space of possible commands, instead of checking for exact matches. eg. "follow the X" might be essential for anything in the world that is X.
2) Do not add every failed command from an issue to ground truth, only if it is within the space of commands we really need in ground truth.
3) Alternatively, we don't add commands to ground truth from model issue trackers, and instead add them to model unit test. Test is not blocking but provides signal into how current model is doing.

## Checklist

TBD, will add subtasks once this is clear

- [ ] Come up with a scalable approach for handling model failures.
- [ ] Implement solution

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.