QAT with trainable=False does not work as expected.

Describe the bug

System information

TensorFlow version (installed from source or binary): 2.6.0

TensorFlow Model Optimization version (installed from source or binary): 0.7.0

Python version: 3.7.10

Describe the expected behavior
After setting trainable=False on layers with a quantisation wrapper applied, the weights in that layer should not change during training.

Describe the current behavior
The loss decreases during training even if all layers are set to be non-trainable.

Code to reproduce the issue

import numpy as np
import tensorflow as tf
import tensorflow_model_optimization as tfmot

A = np.random.uniform(size=(10000, 10, 10))

print('Expected behaviour')
inp = tf.keras.Input(shape=(10, 10), batch_size=10)
out = tf.keras.layers.Dense(10)(inp)
model = tf.keras.Model(inp, out)
for layer in model.layers:
    layer.trainable = False
model.compile(loss='mse')
model.fit(A, A, batch_size=10, epochs=5)
print('{} trainable weights'.format(len(model.layers[1].trainable_weights)))

print('\nQuantised behaviour')
inp = tf.keras.Input(shape=(10, 10), batch_size=10)
out = tfmot.quantization.keras.quantize_annotate_layer(tf.keras.layers.Dense(10))(inp)
quant_model = tfmot.quantization.keras.quantize_apply(tf.keras.Model(inp, out))
for layer in quant_model.layers:
    layer.trainable = False
quant_model.compile(loss='mse')
quant_model.fit(A, A, batch_size=10, epochs=5)
print('{} trainable weights'.format(len(quant_model.layers[2].trainable_weights)))