tensorflow / tensorflow/recommenders

loss is decreasing but val_loss is increasing

Open
#252 2 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2k
Forks
300
PR merge metrics
No merged PRs in 30d

Description

I have followed the tutorial to have the code below. I confused about the result from the code. The loss is decreasing. However, the val_loss is increasing from the beginning. The val_metrics is also decreasing, compared the result from Epoch 5 and Epoch 10.

This is very simple model. Why is it overfitting from the beginning?

This is the result:

Epoch 5:
val_factorized_top_k/top_1_categorical_accuracy: 5.0000e-05 - val_factorized_top_k/top_5_categorical_accuracy: 0.0016 - val_factorized_top_k/top_10_categorical_accuracy: 0.0054 - val_factorized_top_k/top_50_categorical_accuracy: 0.0772 - val_factorized_top_k/top_100_categorical_accuracy: 0.1771 - val_loss: 28846.7070 - val_regularization_loss: 0.0000e+00 - val_total_loss: 28846.7070

Epoch 10:
val_factorized_top_k/top_1_categorical_accuracy: 5.0000e-05 - val_factorized_top_k/top_5_categorical_accuracy: 3.5000e-04 - val_factorized_top_k/top_10_categorical_accuracy: 0.0016 - val_factorized_top_k/top_50_categorical_accuracy: 0.0512 - val_factorized_top_k/top_100_categorical_accuracy: 0.1436 - val_loss: 29914.9844 - val_regularization_loss: 0.0000e+00 - val_total_loss: 29914.9844

Epoch 15:
val_factorized_top_k/top_1_categorical_accuracy: 0.0000e+00 - val_factorized_top_k/top_5_categorical_accuracy: 5.0000e-05 - val_factorized_top_k/top_10_categorical_accuracy: 0.0011 - val_factorized_top_k/top_50_categorical_accuracy: 0.0456 - val_factorized_top_k/top_100_categorical_accuracy: 0.1371 - val_loss: 30586.1602 - val_regularization_loss: 0.0000e+00 - val_total_loss: 30586.1602

from typing import Dict, Text

import tensorflow as tf
import tensorflow_datasets as tfds
import tensorflow_recommenders as tfrs

ratings = tfds.load('movielens/100k-ratings', split="train")
movies = tfds.load('movielens/100k-movies', split="train")

ratings = ratings.map(lambda x: {
    "movie_id": tf.strings.to_number(x["movie_id"]),
    "user_id": tf.strings.to_number(x["user_id"])
})
movies = movies.map(lambda x: tf.strings.to_number(x["movie_id"]))

class Model(tfrs.Model):

  def __init__(self):
    super().__init__()

    # Set up user representation.
    self.user_model = tf.keras.layers.Embedding(
        input_dim=2000, output_dim=64)
    # Set up movie representation.
    self.item_model = tf.keras.layers.Embedding(
        input_dim=2000, output_dim=64)
    # Set up a retrieval task and evaluation metrics over the
    # entire dataset of candidates.
    self.task = tfrs.tasks.Retrieval(
        metrics=tfrs.metrics.FactorizedTopK(
            candidates=movies.batch(128).map(self.item_model)
        )
    )

  def compute_loss(self, features: Dict[Text, tf.Tensor], training=False) -> tf.Tensor:

    user_embeddings = self.user_model(features["user_id"])
    movie_embeddings = self.item_model(features["movie_id"])

    return self.task(user_embeddings, movie_embeddings)


model = Model()
model.compile(optimizer=tf.keras.optimizers.Adagrad(0.1))

# Randomly shuffle data and split between train and test.
tf.random.set_seed(42)
shuffled = ratings.shuffle(100_000, seed=42, reshuffle_each_iteration=False)

train = shuffled.take(80_000)
test = shuffled.skip(80_000).take(20_000)

# Train.
history = model.fit(
    train.batch(4096),
    validation_data=test.batch(4096),
    validation_freq=5,
    epochs=50)

# Evaluate.
model.evaluate(test.batch(4096), return_dict=True)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The entry points are the inline ratings/movies pipeline and Model.compute_loss; reproduce the provided model.fit run first, then inspect the train/test split, embeddings, and Retrieval metrics. Done means identifying and documenting the cause of the diverging validation values, or describing a confirmed correction if the behavior is a defect.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.