How does regularisation affect the training speed? I am getting X100 boost.

Hello!

I am experimenting with the base model from tutorials.
I have noted that after adding tf.keras.regularizers.l2(0.05) ) to embeddings layer two things happen:

1. Metrics start growing - understandable.
2. Speed of trining becomes X50-100 times faster. Why can it happen?

One more thing I noted is that when I add a few Dense layers without any regularizers or add tf.keras.regularizers.l2(1) with big weight to embedding I am getting all accuracies Top1, Top3, ..., Top100 equal to 1, and loss is low and stable. What is happening in this case?

vocab = tf.keras.layers.experimental.preprocessing.StringLookup(vocabulary=vocabulary[slot_name])
embedding = tf.keras.layers.Embedding(vocab.vocabulary_size(), embedding_dim, embeddings_regularizer=tf.keras.regularizers.l2(0.05) )
inputs.append( tf.keras.Sequential([vocab, 
                        embedding, 
                        # tf.keras.layers.Dense(32, activation="relu"),
                        # tf.keras.layers.Dense(32, activation="relu"),
                        ]) )

Image: growing lines on the top - experiments with regularisation. Others - without. top_100_categorical_accuracy

Start from the tutorial base model and compare the shown Embedding configuration with embeddings_regularizer=tf.keras.regularizers.l2(0.05), the larger regularizer, and the added Dense layers. Reproduce the training-speed and Top-k accuracy behavior, checking the available model and metric setup. Done means documenting the cause of the differing results and identifying whether the behavior is expected or indicates an issue.