Calculate brier score for the evaluations table
- Dominant language
- Jupyter Notebook
- Stars
- 201
- Forks
- 62
- PR merge metrics
- No merged PRs in 30d
Description
Brier score gives a sense of calibration (whether the model's predicted probabilities equal observed probabilities, e.g. do 40% of observations with a score of .4 have a 1 label?)
The brier score is the mean squared error between the label and the risk score. If a model has 2000 predictions, it would be 1/2000 sum [(each prediction - each label)^2]
sklearn function: http://scikit-learn.org/stable/modules/model_evaluation.html
Contributor guide
Research direction
Locate the evaluations table and inspect how existing evaluation metrics are calculated and displayed. Read the linked scikit-learn model evaluation documentation and compare it with the stated mean-squared-error formula. Done means the evaluations table includes a Brier score for the model predictions and labels.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100