Model stops learning when using average batch loss instead of sum (Why even use sum?)

I would like to have the average batch loss, instead of the sum. This because it simplifies a lot of things when performing hyper-parameter tuning in regards to learning-rate, regularization parameters and so on.

To do this, I've simply replaced the default task from the source:

    self._loss = loss if loss is not None else tf.keras.losses.CategoricalCrossentropy(
        from_logits=True, reduction=tf.keras.losses.Reduction.SUM)

with the loss (SUM_OVER_BATCH_SIZE ):

task = tfrs.tasks.Retrieval(
  loss=tf.keras.losses.CategoricalCrossentropy(
                from_logits=True, reduction=tf.keras.losses.Reduction.SUM_OVER_BATCH_SIZE                           
            ),
  metrics=tfrs.metrics.FactorizedTopK(
      candidates=movies.batch(128).map(movie_model)
  )
)

When I do this change in the basic retrieval example, the loss never decreases, thus stops learning and top-k performance obviously goes down the drain. I've played around with learning rates and batch sizes but all to no avail.

Because of this, I have two questions:

1. In almost all of neural-network architectures I've seen, the batch-average loss is the norm - so why not for tensorflow-recommenders models?
2. How can I successfully perform the basic retrieval example with batch-average loss?

Start with the basic retrieval example and the shown tfrs.tasks.Retrieval loss configuration; compare SUM with SUM_OVER_BATCH_SIZE while checking the training loss and top-k metrics. Determine why average batch loss prevents learning, then document or reproduce the conditions under which the example trains successfully.