Experiment Validator should catch duplicates in lists
Open
- Dominant language
- Jupyter Notebook
- Stars
- 201
- Forks
- 62
- PR merge metrics
- No merged PRs in 30d
Description
We recently hit an error where the list of percentiles in evaluation config had a duplicate. We could of course squash duplicates in Triage, but I think this would be subpar as it means an incorrect version of the list gets hashed. We should make users fix the config instead.
Contributor guide
Research direction
Locate the Experiment Validator and the evaluation-config validation entry point; first inspect how percentile lists are currently checked. Add coverage for duplicate list values and verify that validation rejects the configuration rather than accepting a list whose contents would be altered before hashing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, tooling
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100