lyst / lyst/lightfm

Handle the item and user features in Dataset

Open
#514 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5.1k
Forks
724
PR merge metrics
No merged PRs in 30d

Description

I've just started working with `lightfm` and have stuck with the `user_features` and `item_features` mapping.
Let me elaborate. I have the following dataframe:
```
df.shape
(25786568, 21)
```
I use the following item and user features correspondingly to fit the model.
```
item_features_list = ['segment_id', 'brand_id', 'is_own_trademark', 'is_alcohol']
user_features_list = ['age', 'gender']
```

1. Firstly I tried to fit dataset in the following way:
```
dataset = Dataset()
dataset.fit(
df['user_id'].unique(),
df['item_id'].unique()),
item_features = df_train_1000[item_features_list],
user_features = df_train_1000[user_features_list]
)
```

That resulted in the following item mapping `dataset._item_feature_mapping`
```
{...
'c14d4e94f4': 410,
'cf3f649a35': 411,
'segment_id': 412,
'brand_id': 413,
'is_own_trademark': 414,
'is_alcohol': 415}
```

That doesn't seem to be the right structure and expectedly that raised an error when I tried to to build the corresponding feature using `dataset.build_item_features(item_features_lightfm)`

```
ValueError: Feature 22 not in feature mapping. Call fit first
```
P.S. 22 is one of the value of `segment_id`

The format of `item_features_lightfm` seems to be correct:
```
`item_features_lightfm = list(zip(df['item_id'].values, df[item_features_list].values))`
```

2. After I went through other [[1](https://github.com/lyst/lightfm/issues/378),[2](https://github.com/lyst/lightfm/issues/330)] issues and [this](https://www.kaggle.com/niyamatalmass/lightfm-hybrid-recommendation-system#Defining-our-necessary-functions) kaggle kernel I came up with the following:

```
dataset = Dataset()
dataset.fit(
df['client_id'].unique(),
df['product_id'].unique(),
item_features=df_train_top[item_features_list].values.flatten(),
user_features=df_train_top[user_features_list].values.flatten()
)
```
This initialization took me ~16 minutes to complete on single machine (2.8 GHz Intel Core i7)

```
item_features_lightfm = list(zip(df['item_id'].values, df[item_features_list].values))
user_features_lightfm = list(zip(df['item_id'].values, df[user_features_list].values))
```
That created the following item mapping `dataset._item_feature_mapping`
```
{...
'3a2cd6476c': 409,
'c14d4e94f4': 410,
'cf3f649a35': 411,
105.0: 412,
177: 413,
0: 414,
119.0: 415,
329: 416,
313.0: 417,
362: 418,
157.0: 419,
226: 420,
212.0: 421,
249: 422,
96.0: 423,
344: 424,
9.0: 425,
320: 426,
'00000': 427,
364: 428,
126.0: 429,
94: 430,
260.0: 431,
169: 432,
....
}
```

Although within this approach I at least managed to compile the code, I don't feel confident enough as the dictionary above creates multiple collisions and creates the heavy dictionary stored in the memory which is crucial for a large datasets. I haven't finished the modeling part yet.

I don't think that I'm on the right path. I'll appreciate if you help me with the proper initialization of the `Datasets` for my case.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Dataset.fit and build_item_features usage shown in the issue, then compare the linked issues and Kaggle example. Check how the dataframe columns are converted into item and user feature inputs, including the separate feature mappings. Done means the mappings are correct, avoid unintended collisions or excessive memory use, and build_item_features no longer raises the reported ValueError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.