dssg / dssg/triage

Calculate brier score for the evaluations table

Open
#207 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
201
Forks
62
PR merge metrics
No merged PRs in 30d

Description

Brier score gives a sense of calibration (whether the model's predicted probabilities equal observed probabilities, e.g. do 40% of observations with a score of .4 have a 1 label?)

The brier score is the mean squared error between the label and the risk score. If a model has 2000 predictions, it would be 1/2000 sum [(each prediction - each label)^2]

sklearn function: http://scikit-learn.org/stable/modules/model_evaluation.html

Contributor guide

Open the contributing guide

Research direction

Locate the evaluations table and inspect how existing evaluation metrics are calculated and displayed. Read the linked scikit-learn model evaluation documentation and compare it with the stated mean-squared-error formula. Done means the evaluations table includes a Brier score for the model predictions and labels.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.