It is unclear why evaluation metrics (auc_score, precision_at_k) fail when train and test data have high overlap
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.1k
- Forks
- 724
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
When I run this code to get my test performance:
```
test_auc = auc_score(model, test_interactions = test_data_matrix, train_interactions = train_data_matrix).mean()
```
I get an error:
> ValueError: Test interactions matrix and train interactions matrix share 745082 interactions. This will cause incorrect evaluation, check your data split.
This is presumably because test_data_matrix is all my data (2.5 years of retail transactions) and training data is all the data except the last 6 months. It is basically failing because train and test have a high degree of overlap.
But why? Surely the whole point of the `train_interactions` argument is so you can exclude the overlap? Shouldn't this be a warning rather than an error that fails the whole function?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the auc_score evaluation call and inspect how train_interactions and test_interactions are compared when the matrices contain overlapping retail transactions. Reproduce the reported 745082-interaction overlap, then determine whether the current error behavior matches the documented evaluation contract and what outcome should count as done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100