dssg / dssg/triage

Better handling of multiple prediction windows

Open
#327 0 comments 0 reactions 0 assignees View on GitHub
audition
Dominant language
Jupyter Notebook
Stars
201
Forks
62
PR merge metrics
No merged PRs in 30d

Description

_From @shaycrk on October 17, 2017 21:26_

Audition currently considers the first prediction window/set of evaluations after the training end date, but we may want to use the information contained in multiple prediction windows for model selection. This would require some considerable refactoring to:
* ensure that all train end times have a comparable set of subsequent prediction windows to evaluate across
* include additional prediction windows in the distance from best calculations
* develop metrics and visualizations for studying model group stability
* pass additional prediction window information to selection rules to allow for rules that use stability as a factor in model selection

_Copied from original issue: dssg/audition#24_

Contributor guide

Open the contributing guide

Research direction

Start by mapping how training end times, prediction windows, distance-from-best calculations, and selection rules are represented in the project. Review the existing evaluation and visualization entry points, then define comparable windows, stability metrics, visualizations, and the information selection rules would need before implementation can begin.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-visualization, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.