tensorflow / tensorflow/recommenders

ScaNN evaluation gives better metrics than BruteForce evaluation

Open
#264 3 comments 1 reaction 1 assignee View on GitHub

@maciejkula is already working on this.

Since Mar 24, 2021.

question
Dominant language
Python
Stars
2k
Forks
300
PR merge metrics
No merged PRs in 30d

Description

Following the documentation (https://www.tensorflow.org/recommenders/examples/efficient_serving#evaluating_the_approximation), I am trying to compare the performance the ScaNN evaluation and BruteForce evaluation.

Code for BruteForce evaluation:

item_embeddings = items_data.batch(batch_size).map(model.candidate_model)
# Override the existing streaming candidate source and turn on the compute_metrics
model.task.factorized_metrics = tfrs.metrics.FactorizedTopK(
    candidates=item_embeddings
)
model.compute_metrics = True
# Need to recompile the model for the changes to take effect.
model.compile()
%time results_eval = model.evaluate(cached_test, return_dict=True)
results_eval

which gives me result:
{'factorized_top_k/top_1_categorical_accuracy': 0.0016201249090954661, 'factorized_top_k/top_5_categorical_accuracy': 0.006863438058644533, 'factorized_top_k/top_10_categorical_accuracy': 0.01172381266951561, 'factorized_top_k/top_50_categorical_accuracy': 0.03605514392256737, 'factorized_top_k/top_100_categorical_accuracy': 0.05449511110782623, 'loss': 9786.341796875, 'regularization_loss': 0, 'total_loss': 9786.341796875}

Code for ScaNN evaluation:

scann = tfrs.layers.factorized_top_k.ScaNN(num_reordering_candidates=1000)
scann.index(item_embeddings)
model.task.factorized_metrics = tfrs.metrics.FactorizedTopK(
    candidates=scann
)
model.compute_metrics = True
model.compile()
%time scann_result = model.evaluate(cached_test, return_dict=True)
scann_result

And I get:
{'factorized_top_k/top_1_categorical_accuracy': 0.004065040498971939, 'factorized_top_k/top_5_categorical_accuracy': 0.012725344859063625, 'factorized_top_k/top_10_categorical_accuracy': 0.019765524193644524, 'factorized_top_k/top_50_categorical_accuracy': 0.05107811838388443, 'factorized_top_k/top_100_categorical_accuracy': 0.07484976947307587, 'loss': 9786.341796875, 'regularization_loss': 0, 'total_loss': 9786.341796875}

Is there any good reason that the ANN approach is performing better than a BruteForce approach?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.