Hybrid model have lower Precision@K compare to pure CF
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.1k
- Forks
- 724
- PR merge metrics
- No merged PRs in 30d
Description
Hi Maciej,
I'm testing LightFM for my recommendation system in e-commerce of grocery products (everything you could buy in a convenient store). I've tested LightFM hybrid to pure collarborative filtering (also LightFM, just without users and item features) and got smaller preciesion@10. I've read your paper and it seems to point out that hybrid model should outperform pure CF model, but my experiment get the opposite results.
Here are the descriptions of my approach:
## Dataset
* Interaction matrix: 9915 x 17199; 98% sparsity (or ~2% density)
* Purchase data of 9915 users across 17199 items.
* All users have at least 1 transaction during the sample period
* User features matrix: 9915 x 9930
* Including an identity matrix and 15 additional features on age, gender, geographic regions, etc
* Item features matrix: 17199 x 21007
* Including an identity matrix and 3808 features based on brand name, categories, product descriptions
## Implementation:
1. Train-Test data split by timestamp (because 1 user might re-purchase an item) then interaction matrix was built using `lighfm.data.build_interaction()`
2. Training and evaluation:
```
model = LightFM(loss='warp',
no_components=80,
item_alpha= 1e-7,
learning_rate = 0.02,
max_sampled = 50)
```
* Hybrid model
```
model_hybird = model.fit(train,
item_features=item_features,
user_features = user_features,
epochs = 80,
num_threads = 4)
test_precision = precision_at_k(model_hybrid, test, item_features = item_features,
user_features = user_features, num_threads = 4, k= 10).mean()
```
* Pure CF model
```
model_simple = model.fit(train,
epochs = 80,
num_threads = 4)
test_precision = precision_at_k(model_simple, test, num_threads = 4, k= 10).mean()
```
3. Results: hybrid_precision@10 = 0.057814, pureCF_precision@10 = 0.070189. I've tried several things to try to increase the test precision of hybrid model including: Use weight matrix for training, optimize hyper parameter using grid search, normalize user features and item features, calculate weight for user features and item features with TFIDF. But so far the results always have pure CF outperform the hybrid model.
### Would appreciate any advice on this. Thank you!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the reported calls to lighfm.data.build_interaction(), LightFM.fit(), and precision_at_k(), then compare the hybrid and pure collaborative-filtering evaluation paths. Determine whether the observed Precision@10 difference is expected for this dataset and configuration, and document the explanation or required corrective guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100