tensorflow / tensorflow/recommenders
ScaNN evaluation gives better metrics than BruteForce evaluation
@maciejkula is already working on this.
Since Mar 24, 2021.
- Dominant language
- Python
- Stars
- 2k
- Forks
- 300
- PR merge metrics
- No merged PRs in 30d
Description
Following the documentation (https://www.tensorflow.org/recommenders/examples/efficient_serving#evaluating_the_approximation), I am trying to compare the performance the ScaNN evaluation and BruteForce evaluation.
Code for BruteForce evaluation:
item_embeddings = items_data.batch(batch_size).map(model.candidate_model)
# Override the existing streaming candidate source and turn on the compute_metrics
model.task.factorized_metrics = tfrs.metrics.FactorizedTopK(
candidates=item_embeddings
)
model.compute_metrics = True
# Need to recompile the model for the changes to take effect.
model.compile()
%time results_eval = model.evaluate(cached_test, return_dict=True)
results_eval
which gives me result:
{'factorized_top_k/top_1_categorical_accuracy': 0.0016201249090954661,
'factorized_top_k/top_5_categorical_accuracy': 0.006863438058644533,
'factorized_top_k/top_10_categorical_accuracy': 0.01172381266951561,
'factorized_top_k/top_50_categorical_accuracy': 0.03605514392256737,
'factorized_top_k/top_100_categorical_accuracy': 0.05449511110782623,
'loss': 9786.341796875,
'regularization_loss': 0,
'total_loss': 9786.341796875}
Code for ScaNN evaluation:
scann = tfrs.layers.factorized_top_k.ScaNN(num_reordering_candidates=1000)
scann.index(item_embeddings)
model.task.factorized_metrics = tfrs.metrics.FactorizedTopK(
candidates=scann
)
model.compute_metrics = True
model.compile()
%time scann_result = model.evaluate(cached_test, return_dict=True)
scann_result
And I get:
{'factorized_top_k/top_1_categorical_accuracy': 0.004065040498971939,
'factorized_top_k/top_5_categorical_accuracy': 0.012725344859063625,
'factorized_top_k/top_10_categorical_accuracy': 0.019765524193644524,
'factorized_top_k/top_50_categorical_accuracy': 0.05107811838388443,
'factorized_top_k/top_100_categorical_accuracy': 0.07484976947307587,
'loss': 9786.341796875,
'regularization_loss': 0,
'total_loss': 9786.341796875}
Is there any good reason that the ANN approach is performing better than a BruteForce approach?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.