Item and User Normalization
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.1k
- Forks
- 724
- PR merge metrics
- No merged PRs in 30d
Description
I trained my LightFM model on user and item features along with interactions. I noticed some things which didnt make sense to me, So I'm hoping someone would make me understand. I am using the following code for splitting, training and evaluation:
```
train, test = random_train_test_split(interactions, test_percentage=0.2, random_state=np.random.RandomState(5))
train_weights, test_weights = random_train_test_split(weights, test_percentage=0.2, random_state=np.random.RandomState(5))
#model training
model.fit(train,
user_features=user_features,
item_features= item_features,
sample_weight= train_weights,
epochs=10)
# Evaluate the model on the test set using auc
auc = auc_score(model,
test,
user_features=user_features,
item_features=item_features,
).mean()
```
The things I need to understand is the effect of user and item normalization on the evaluation. Following is the code :
```
user_features = dataset.build_user_features(User_df['features'].tolist(), normalize= True)
item_features = dataset.build_item_features(Product_df['features'].tolist(), normalize= True)
```
Things which are confusing:
- When I put normalize =False for both, then I get AUC approx 84%.
- When I put normalize =True for both, then I get AUC approx 99%.
- When I normalize only user features, the AUC is still 99%.
- When I normalize only item features, the AUC is still 99%.
- When I put normalize =False for user and exclude item features from training, the AUC is still 99%.
- When I put normalize =False for user and exclude user features from training, the AUC is 84%.
I'd like to know if normalization can have this much effect and if above scenarios make any sense. Also additionally, AUC of 99% seems too good to be true.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported cases using random_train_test_split, build_user_features, build_item_features, model.fit, and auc_score with the code in the issue. Read the feature-building and evaluation entry points to determine how normalization and excluded features affect AUC. Done means documenting whether the results are expected and explaining the unusually high score.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100