dssg / dssg/eights

Experiment tree interface tweaking

Open
#18 0 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
14
Forks
2
PR merge metrics
No merged PRs in 30d

Description

At the CV level of trial, the subset level of trial, and at the trial level, we should be able to:

1) Average runs/subsets/trials together. Average score, average ROC curve etc. This makes sense if and only if the runs are indistinguishable (i.e. KFold cross validation)

2) Take the metric for the best run/subset/trial. This makes sense if averaging things together is nonsense (i.e. sliding window cross validation.)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the experiment-tree interface and the code that computes or displays metrics at CV, subset, and trial levels. Review how KFold and sliding-window runs are represented, then define how averaging and best-run selection should appear, including score and ROC-curve behavior. Done means each level supports the appropriate aggregation without combining indistinguishable and distinguishable runs incorrectly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.