dssg / dssg/police-eis

Add absolute accuracy

Open
#168 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
56
Forks
19
PR merge metrics
No merged PRs in 30d

Description

Webapp should plot predicted versus true probabilities. Bin predicted probabilities, e.g. all between .4 and .5 and see if the average is ~.45. I'm looking for something like [this](http://d39mdc1iz3oyyf.cloudfront.net/content/erj/39/2/366/F2.large.jpg).

We can also get an AUC score for this, but unlike ROC, the best is 0.5 and the worst is 1.0 or 0.0. We should generate this AUC score for every model and store it in the `results.evaluations` table

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the webapp's model-evaluation flow and the results.evaluations table, then trace how prediction outputs are currently presented. Confirm the requested probability binning and AUC convention from the issue. Done means the webapp shows predicted versus true probabilities and stores the requested score for every model.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-visualization, databases, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.