Store ranks used for metrics in predictions table
- Dominant language
- Jupyter Notebook
- Stars
- 201
- Forks
- 62
- PR merge metrics
- No merged PRs in 30d
Description
when calculating metrics, triage resolves ties in scores using a random selection (e.g., if calculating precision at top 5 and the first 7 scores are [.99, .98, .97. .96, .95, .95. .95], triage will randomly select one of the .95s to include). To preserve information about the exact set of entities included in the top k, it would be good to:
1. rank all of the entities by scores (e.g., [.22, .34, .24, .12, .12, .12] becomes [4, 6, 5, 1, 1, 1])
2. fill in tied ranks with all of the integer ranks they cover (e.g., replace the rank 1s in the previous example with 1, 2, and 3, assigned randomly to the three entities sharing the first rank)
3. save these ranks in results.predictions under the name "ranks_with_ties_randomly_broken" or somesuch
this way, the same ranks can be used for postmodeling analysis of the list.
Contributor guide
Research direction
Start by tracing the metric-calculation path and the results.predictions data described in the issue. Confirm how ties are currently selected, then verify that ranks with ties randomly broken are stored and can be reused for postmodeling analysis.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100