Handle the item and user features in Dataset
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.1k
- Forks
- 724
- PR merge metrics
- No merged PRs in 30d
Description
I've just started working with `lightfm` and have stuck with the `user_features` and `item_features` mapping.
Let me elaborate. I have the following dataframe:
```
df.shape
(25786568, 21)
```
I use the following item and user features correspondingly to fit the model.
```
item_features_list = ['segment_id', 'brand_id', 'is_own_trademark', 'is_alcohol']
user_features_list = ['age', 'gender']
```
1. Firstly I tried to fit dataset in the following way:
```
dataset = Dataset()
dataset.fit(
df['user_id'].unique(),
df['item_id'].unique()),
item_features = df_train_1000[item_features_list],
user_features = df_train_1000[user_features_list]
)
```
That resulted in the following item mapping `dataset._item_feature_mapping`
```
{...
'c14d4e94f4': 410,
'cf3f649a35': 411,
'segment_id': 412,
'brand_id': 413,
'is_own_trademark': 414,
'is_alcohol': 415}
```
That doesn't seem to be the right structure and expectedly that raised an error when I tried to to build the corresponding feature using `dataset.build_item_features(item_features_lightfm)`
```
ValueError: Feature 22 not in feature mapping. Call fit first
```
P.S. 22 is one of the value of `segment_id`
The format of `item_features_lightfm` seems to be correct:
```
`item_features_lightfm = list(zip(df['item_id'].values, df[item_features_list].values))`
```
2. After I went through other [[1](https://github.com/lyst/lightfm/issues/378),[2](https://github.com/lyst/lightfm/issues/330)] issues and [this](https://www.kaggle.com/niyamatalmass/lightfm-hybrid-recommendation-system#Defining-our-necessary-functions) kaggle kernel I came up with the following:
```
dataset = Dataset()
dataset.fit(
df['client_id'].unique(),
df['product_id'].unique(),
item_features=df_train_top[item_features_list].values.flatten(),
user_features=df_train_top[user_features_list].values.flatten()
)
```
This initialization took me ~16 minutes to complete on single machine (2.8 GHz Intel Core i7)
```
item_features_lightfm = list(zip(df['item_id'].values, df[item_features_list].values))
user_features_lightfm = list(zip(df['item_id'].values, df[user_features_list].values))
```
That created the following item mapping `dataset._item_feature_mapping`
```
{...
'3a2cd6476c': 409,
'c14d4e94f4': 410,
'cf3f649a35': 411,
105.0: 412,
177: 413,
0: 414,
119.0: 415,
329: 416,
313.0: 417,
362: 418,
157.0: 419,
226: 420,
212.0: 421,
249: 422,
96.0: 423,
344: 424,
9.0: 425,
320: 426,
'00000': 427,
364: 428,
126.0: 429,
94: 430,
260.0: 431,
169: 432,
....
}
```
Although within this approach I at least managed to compile the code, I don't feel confident enough as the dictionary above creates multiple collisions and creates the heavy dictionary stored in the memory which is crucial for a large datasets. I haven't finished the modeling part yet.
I don't think that I'm on the right path. I'll appreciate if you help me with the proper initialization of the `Datasets` for my case.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Dataset.fit and build_item_features usage shown in the issue, then compare the linked issues and Kaggle example. Check how the dataframe columns are converted into item and user feature inputs, including the separate feature mappings. Done means the mappings are correct, avoid unintended collisions or excessive memory use, and build_item_features no longer raises the reported ValueError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100