Basic Recommender Ranking: Big Optimization issue

Hi everyone,

I am a Data Scientist and I start following your examples on Tensorflow recommenders few weeks ago.

I noticed that the Basic Recommender (ranking) is deeply flawed.

Here is the reason:

1. Try to plot the ranking results as histogram or kde
2. Do the same with the labels from the test dataset
3. Plot also a random integer distribution (from 1 to 5).

If you add the following Code to the basic ranking recommender notebook. You should get my same results.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

sns.set_style('darkgrid')

res_data = pd.DataFrame()

# get predictions from cached test 
res_data['predictions'] = model.predict(cached_test)[:, 0]

# add random integer distribution
res_data['random'] = np.random.randint(1,6, len(res_data))

# get user rating from test dataset
test_labels = []
for r in cached_test:
  test_labels.append((r['user_rating']).numpy())

res_data['test_labels'] = np.concatenate(test_labels)

# plot everythin as kde
plt.figure(figsize=(10,7), dpi=100)
sns.kdeplot(data=res_data, fill=True, bw_adjust=0.9, alpha=0.6, linewidth=0, legend=False)
plt.legend(["Predictions", "Random Monkey",  "Test Labels"][::-1], title="Legend", fontsize=12, title_fontsize=16)
plt.title('Predictions vs. test labels', fontsize=20);

Results:

Did you get the issue?
Is it normal that our predictions are not able to properly rank the input data since they are distributed in a gaussian way around a mean value of 3.5?
Did I miss something?

Thank you in advance!
@albertvillanova @maciejkula @MarkDaoust @hojinYang

Start with docs/examples/basic_ranking.ipynb and reproduce the reported comparison using model.predict(cached_test), the test labels, and the random distribution. Determine whether the prediction shape reflects a flaw in the ranking example; done means documenting a clear conclusion and identifying the required example or model change.