lyst / lyst/lightfm

Does calculating precision@k for one user makes sense?

Open
#707 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.1k
Forks
724
PR merge metrics
No merged PRs in 30d

Description

Hi I am building a "product-product recommender system", where there is one main product, and it may be related to multiple addon products.

I am creating an interaction matrix based on relevance b/w my main prod and addon prod, hence a binary matrix where if the products are related in actual data, the value is 1, otherwise 0.

I have a sparse matrix situation where number of non-zero entries is much much less as compared to zero entries, which is fine because I only need few relevant items for every main product, to train my model.
Screenshot 2024-02-22 at 3 24 16 PM

**Problem:**
I am having difficulty in understanding what precision and recall value is good for me. I want _my precision to be high_ in-terms of recommending most relevant top few products whenever a main product on the website is selected.

1. If I take a subset of 10 values from my interactions as well as main product features, and I try to check precision and recall for these 10 values against all addon product. Does that make sense?

`val_interactions_subset = val_interactions.tocsr()[:10, :].tocoo()`
`product_features_subset = product_features[:10, :]`
`addon_product_features_subset = addon_product_features`

For this I get a _precision@k=3_ of 0.2857 and _recall@k=2_ of 0.3785 for this. My understanding is, my model is right on avg 28% of the times it predicts addon product for these 10 main product. But this is very low, however my train interactions were of size (165480, 28576), from which I split only 15% into validation interactions.

2. Now, for analysis purpose: **I want to check precision and recall for each main product in validation set individually, like how well my model is able to capture relevance b/w that one main product and its corresponding addon products. Is this the sound thing to do?** I am not able to do this with _`lightfm.evaluations.precision@k`_ function for one main product only.

I need help with this quickly. Please if someone can help me with understanding this .

Thanks in advance.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with lightfm.evaluations.precision@k and the shown validation-subset calls. Clarify whether evaluating one main product at a time is supported, what the precision and recall values should represent for this interaction matrix, and whether the requested behavior needs an API or documentation change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.