dssg / dssg/triage

Experiment Validator should catch duplicates in lists

Open
#462 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
201
Forks
62
PR merge metrics
No merged PRs in 30d

Description

We recently hit an error where the list of percentiles in evaluation config had a duplicate. We could of course squash duplicates in Triage, but I think this would be subpar as it means an incorrect version of the list gets hashed. We should make users fix the config instead.

Contributor guide

Open the contributing guide

Research direction

Locate the Experiment Validator and the evaluation-config validation entry point; first inspect how percentile lists are currently checked. Add coverage for duplicate list values and verify that validation rejects the configuration rather than accepting a list whose contents would be altered before hashing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, tooling
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.