Better handling of multiple prediction windows
- Dominant language
- Jupyter Notebook
- Stars
- 201
- Forks
- 62
- PR merge metrics
- No merged PRs in 30d
Description
_From @shaycrk on October 17, 2017 21:26_
Audition currently considers the first prediction window/set of evaluations after the training end date, but we may want to use the information contained in multiple prediction windows for model selection. This would require some considerable refactoring to:
* ensure that all train end times have a comparable set of subsequent prediction windows to evaluate across
* include additional prediction windows in the distance from best calculations
* develop metrics and visualizations for studying model group stability
* pass additional prediction window information to selection rules to allow for rules that use stability as a factor in model selection
_Copied from original issue: dssg/audition#24_
Contributor guide
Research direction
Start by mapping how training end times, prediction windows, distance-from-best calculations, and selection rules are represented in the project. Review the existing evaluation and visualization entry points, then define comparable windows, stability metrics, visualizations, and the information selection rules would need before implementation can begin.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-visualization, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100