lyst / lyst/lightfm

It is unclear why evaluation metrics (auc_score, precision_at_k) fail when train and test data have high overlap

Open
#658 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5.1k
Forks
724
PR merge metrics
No merged PRs in 30d

Description

Hi,

When I run this code to get my test performance:
```
test_auc = auc_score(model, test_interactions = test_data_matrix, train_interactions = train_data_matrix).mean()
```
I get an error:

> ValueError: Test interactions matrix and train interactions matrix share 745082 interactions. This will cause incorrect evaluation, check your data split.

This is presumably because test_data_matrix is all my data (2.5 years of retail transactions) and training data is all the data except the last 6 months. It is basically failing because train and test have a high degree of overlap.

But why? Surely the whole point of the `train_interactions` argument is so you can exclude the overlap? Shouldn't this be a warning rather than an error that fails the whole function?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the auc_score evaluation call and inspect how train_interactions and test_interactions are compared when the matrices contain overlapping retail transactions. Reproduce the reported 745082-interaction overlap, then determine whether the current error behavior matches the documented evaluation contract and what outcome should count as done.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.