tensorflow / tensorflow/recommenders
[Question] Ranking returning always same order
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2k
- Forks
- 300
- PR merge metrics
- No merged PRs in 30d
Description
hello, there!
I was exploring the basic ranking tutorial and it seems that, for a fixed set of items, the ranking values vary very little when comparing different users and even cold-start users. At first, I thought this was happening because the retrieval step would be responsible for selecting the "right" items to be ranked, but then I manually selected items that a specific user gave a high score, and yet the model would give the same ranking for both this user and a cold start user.
test_movie_titles = [
"M*A*S*H (1970)", "Dances with Wolves (1990)", "Speed (1994)",
"Swingers (1996)", "Jurassic Park (1993)", "In the Line of Fire (1993)"
]
test_ratings = {}
for movie_title in test_movie_titles:
test_ratings[movie_title] = model({
"user_id": np.array(["276"]),
"movie_title": np.array([movie_title])
})
print("Ratings:")
for title, score in sorted(test_ratings.items(), key=lambda x: x[1], reverse=True):
print(f"{title}: {score}")
regular user ("276")
Ratings:
M*A*S*H (1970): [[3.9895713]]
Swingers (1996): [[3.8475957]]
In the Line of Fire (1993): [[3.7985928]]
Dances with Wolves (1990): [[3.773054]]
Jurassic Park (1993): [[3.7534351]]
Speed (1994): [[3.677456]]
cold start user ("")
Ratings:
M*A*S*H (1970): [[3.70247]]
Swingers (1996): [[3.5549703]]
In the Line of Fire (1993): [[3.508114]]
Dances with Wolves (1990): [[3.480867]]
Jurassic Park (1993): [[3.4616296]]
Speed (1994): [[3.387645]]
I trained the model using some suggestions given on https://github.com/tensorflow/recommenders/issues/591 to make sure the model was not undertrained, and also added more movie titles to the test, but the result is consistent.
Is this an expected behavior or did I miss something in the tutorial? I imagined the ranking would have a substantial difference between users.
Thanks in advance!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the basic ranking tutorial and reproduce the provided model calls for user "276" and the cold-start user. Compare the ranking inputs and outputs with the tutorial and the training changes described in issue #591; done means determining whether the similar rankings are expected or identifying a specific tutorial or model problem.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100