mlcommons / mlcommons/modelbench
Cheval proxy can exclude annotators
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 134
- Forks
- 36
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 17
Description
Cheval assumes that any instances created with cheval can be used by the proxy. But sometimes we want to spin up annotators that are left out of the pool. For this card, add a way to exclude annotators from proxy usage and some command-line ways to create those instances and change their statuses.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Cheval proxy, annotator creation, and command-line entry points. Trace how annotators enter the proxy pool and how their statuses are changed. Done means annotators can be explicitly excluded from proxy usage and command-line options support creating those instances and changing their statuses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100