Including item features seems to reduce performance.

First, it is totally possible that I am misunderstanding something basic or have a bug in my code.

But I am consistently finding that adding item features actually _reduces_ performance compared with collaborative filtering.

I first did the analysis on some internal data, but reproduced it with a public example to share here. Here is a notebook with an example on goodreads data: https://github.com/jemmott/lightfm-goodbooks-debug

The punch line is I looked an implicit example - trying to predict if a user will rate a book. I used mean reciprocal rank (MRR) as the metric, but results were similar for R@K and P@K. Performance is significantly reduced when I include authors as an item feature when compared with no item features (pure collaborative filtering). I did not explore user features.

On a hunch I decided to test something kind of strange. I decided to shuffle the item features - permuting them so that they are randomly assigned to each item. I then trained and cross validated LightFM, and found the change in MRR. I repeated that 100 times, and drew a histogram of the results, shown in blue below. The x axis is percent change from CF. The red line is the result with the actual item assignments.


What this tells me is that not only are the item features actually reducing the performance, but the actual (non shuffled) item features are on average no better at predicting than randomly shuffled ones. This seems really bad.

I also tried an example where I included the item ids as item features to add the identity matrix back in. Performance was still worse than pure CF (no features), but did improve slightly.

It seems like I am not alone with this - here are two other examples of people seeing worse performance when adding item features:

- https://www.ethanrosenthal.com/2016/11/07/implicit-mf-part-2/
- https://towardsdatascience.com/recommendation-system-part-1-use-of-collaborative-filtering-and-hybrid-collaborative-content-in-6137ba64ad58

Anyone know what is going on here?

Start with the linked goodreads notebook and reproduce the comparison between collaborative filtering and models using author or shuffled item features. Check the MRR, R@K, and P@K evaluation setup and the item-feature configuration before investigating the implementation. Done means explaining whether the degradation is expected or identifying a reproducible implementation problem.

Written by the indexing model from the issue text.