lyst / lyst/lightfm

Evaluating precision@k/recall@k for users with <k "known positives"

Open
#529 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.1k
Forks
724
PR merge metrics
No merged PRs in 30d

Description

Hi,

I have a question about the precision@k and recall@k evaluation functions. I can see from the source code that precision is calculated as:

precision = np.squeeze(np.array(ranks.sum(axis=1))) / k

Wouldn't this calculation underestimate performance if I have a users who have voted/rated

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the precision@k and recall@k evaluation functions and the shown ranks.sum(axis=1) / k calculation. Determine the expected behavior for users with fewer than k known positives and whether the current evaluation matches that definition; done means the behavior has a documented resolution or a targeted regression test if it is incorrect.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.