dssg / dssg/triage

Validate model groups all have results for same/full set of end times

Open
#326 1 comment 0 reactions 0 assignees View on GitHub
audition
Dominant language
Jupyter Notebook
Stars
201
Forks
62
PR merge metrics
No merged PRs in 30d

Description

_From @shaycrk on August 15, 2017 20:17_

If the model groups we're considering haven't all been tested on the same set of test sets, we might not be making fair comparisons for model selection, so we should check the input to ensure the a uniform set of test sets are used throughout.

_Copied from original issue: dssg/audition#18_

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Locate the input handling for model groups and the existing validation tests, then trace how test sets or end times are represented. Done means inputs with differing test sets are detected so model comparisons use a uniform set throughout.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.