lyst / lyst/lightfm

Item and User Normalization

Open
#697 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5.1k
Forks
724
PR merge metrics
No merged PRs in 30d

Description

I trained my LightFM model on user and item features along with interactions. I noticed some things which didnt make sense to me, So I'm hoping someone would make me understand. I am using the following code for splitting, training and evaluation:

```
train, test = random_train_test_split(interactions, test_percentage=0.2, random_state=np.random.RandomState(5))
train_weights, test_weights = random_train_test_split(weights, test_percentage=0.2, random_state=np.random.RandomState(5))

#model training
model.fit(train,
user_features=user_features,
item_features= item_features,
sample_weight= train_weights,
epochs=10)

# Evaluate the model on the test set using auc
auc = auc_score(model,
test,
user_features=user_features,
item_features=item_features,
).mean()
```
The things I need to understand is the effect of user and item normalization on the evaluation. Following is the code :

```
user_features = dataset.build_user_features(User_df['features'].tolist(), normalize= True)

item_features = dataset.build_item_features(Product_df['features'].tolist(), normalize= True)
```
Things which are confusing:

- When I put normalize =False for both, then I get AUC approx 84%.
- When I put normalize =True for both, then I get AUC approx 99%.
- When I normalize only user features, the AUC is still 99%.
- When I normalize only item features, the AUC is still 99%.
- When I put normalize =False for user and exclude item features from training, the AUC is still 99%.
- When I put normalize =False for user and exclude user features from training, the AUC is 84%.

I'd like to know if normalization can have this much effect and if above scenarios make any sense. Also additionally, AUC of 99% seems too good to be true.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported cases using random_train_test_split, build_user_features, build_item_features, model.fit, and auc_score with the code in the issue. Read the feature-building and evaluation entry points to determine how normalization and excluded features affect AUC. Done means documenting whether the results are expected and explaining the unusually high score.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.