dssg / dssg/triage

Store ranks used for metrics in predictions table

Open
#347 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
201
Forks
62
PR merge metrics
No merged PRs in 30d

Description

when calculating metrics, triage resolves ties in scores using a random selection (e.g., if calculating precision at top 5 and the first 7 scores are [.99, .98, .97. .96, .95, .95. .95], triage will randomly select one of the .95s to include). To preserve information about the exact set of entities included in the top k, it would be good to:

1. rank all of the entities by scores (e.g., [.22, .34, .24, .12, .12, .12] becomes [4, 6, 5, 1, 1, 1])
2. fill in tied ranks with all of the integer ranks they cover (e.g., replace the rank 1s in the previous example with 1, 2, and 3, assigned randomly to the three entities sharing the first rank)
3. save these ranks in results.predictions under the name "ranks_with_ties_randomly_broken" or somesuch

this way, the same ranks can be used for postmodeling analysis of the list.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the metric-calculation path and the results.predictions data described in the issue. Confirm how ties are currently selected, then verify that ranks with ties randomly broken are stored and can be reused for postmodeling analysis.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.