tensorflow / tensorflow/recommenders
invalid results on during evaluation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2k
- Forks
- 300
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I am following this tutorial https://www.tensorflow.org/recommenders/examples/basic_retrieval .
during model.fit and model.evaluate i am passing the same dataset but i am getting very different numbers like below.
after model.evaluate results:
{'factorized_top_k/top_1_categorical_accuracy': 6.989097164478153e-05, 'factorized_top_k/top_5_categorical_accuracy': 0.0002096729149343446, 'factorized_top_k/top_10_categorical_accuracy': 0.0003494548436719924, 'factorized_top_k/top_50_categorical_accuracy': 0.000629018759354949, 'factorized_top_k/top_100_categorical_accuracy': 0.0011182555463165045, 'loss': 14674.890625, 'regularization_loss': 0, 'total_loss': 14674.890625}
during training results:
Epoch 3/10
1/2 [==============>...............] - ETA: 2s - factorized_top_k/top_1_categorical_accuracy: 0.4688 - factorized_top_k/top_5_categorical_accuracy: 0.4733 - factorized_top_k/top_10_categorical_accuracy: 0.4733 - factorized_top_k/top_50_categorical_accuracy: 0.4745 - factorized_top_k/top_100_categorical_accuracy: 0.4752 - loss: 19850.7344 - regularization_loss: 0.0000e+00 - total_loss: 19852/2 [==============================] - ETA: 0s - factorized_top_k/top_1_categorical_accuracy: 0.6085 - factorized_top_k/top_5_categorical_accuracy: 0.6209 - factorized_top_k/top_10_categorical_accuracy: 0.6273 - factorized_top_k/top_50_categorical_accuracy: 0.6323 - factorized_top_k/top_100_categorical_accuracy: 0.6348 - loss: 17292.4678 - regularization_loss: 0.0000e+00 - total_loss: 17292/2 [==============================] - 8s 6s/step - factorized_top_k/top_1_categorical_accuracy: 0.6085 - factorized_top_k/top_5_categorical_accuracy: 0.6209 - factorized_top_k/top_10_categorical_accuracy: 0.6273 - factorized_top_k/top_50_categorical_accuracy: 0.6323 - factorized_top_k/top_100_categorical_accuracy: 0.6348 - loss: 16439.7122 - regularization_loss: 0.0000e+00 - total_loss: 16439.7122 - val_factorized_top_k/top_1_categorical_accuracy: 0.0000e+00 - val_factorized_top_k/top_5_categorical_accuracy: 0.0000e+00 - val_factorized_top_k/top_10_categorical_accuracy: 0.0000e+00 - val_factorized_top_k/top_50_categorical_accuracy: 0.0000e+00 - val_factorized_top_k/top_100_categorical_accuracy: 0.0000e+00 - val_loss: 14709.7910 - val_regularization_loss: 0.0000e+00 - val_total_loss: 14709.7910
i kept all train-val-test dataset as same .
`batch_size = config.BATCH_SIZE
S = num_samples
train_sz = int(S*0.8)
val_sz = S - train_sz
shuffled_train = ratings.shuffle(S ,seed=seed, reshuffle_each_iteration=True)
train = shuffled_train.take(train_sz)
val = shuffled_train.skip(train_sz).take(val_sz)
cached_train = train.shuffle(S).batch(batch_size).cache()
cached_val = val.shuffle(S).batch(batch_size).cache()
cached_test = ratings_test.batch(batch_size).cache()`
model.fit(cached_train, epochs=config.EPOCHS,validation_data=cached_train)
model.evaluate(cached_train, return_dict=True)
i would appreciate your help
thanks
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the TensorFlow Recommenders basic retrieval tutorial and compare the model.fit call using validation_data=cached_train with model.evaluate(cached_train). Inspect the ratings shuffle, take/skip split, batching, caching, and repeated dataset use shown in the report. Done means identifying why the training and evaluation metrics differ and recording a reproducible explanation or correction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100