Resnet50 pretrained model for fine tuning, the model is not convergence

Hi,

I'm using pretrained Resnet50 model for my own data's training. The model is not convergence even the train accuracy looks good shown by the log, and validation loss and accuracy is not improved during the training phase.

I also test the trained model on training and val set, the accuracy is very pool(see below)

And I try the tensorflow 1.15.0 and 2.4.0 different version, the problem is the same. Then I just change to VGG model, it works fine(no convergence problem). So could help on this issue?

My code is :
`

base_model = tf.keras.applications.ResNet50(include_top=False)
base_model.trainable = False

model = tf.keras.models.Sequential([
    base_model,
    #tf.keras.layers.Conv2D(filters=num_cat, kernel_size=1),
    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Dense(units=num_cat)
])
model.summary()
optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)
loss_func = tf.keras.losses.CategoricalCrossentropy(from_logits=True)
eval_func = tf.keras.metrics.CategoricalAccuracy()

model.compile(
    optimizer=optimizer,
    loss=loss_func,
    metrics=[eval_func]
)
history = model.fit(train_ds, epochs=10, validation_data=val_ds)
model.save_weights('./checkpoints/final')`

And the training log is:

Train on 156 steps, validate on 39 steps
Epoch 1/10
156/156 [==============================] - 183s 1s/step - loss: 1.8214 - categorical_accuracy: 0.5048 - val_loss: 3.5411 - val_categorical_accuracy: 0.0386
Epoch 2/10
156/156 [==============================] - 33s 211ms/step - loss: 0.4409 - categorical_accuracy: 0.8931 - val_loss: 3.6663 - val_categorical_accuracy: 0.0386
Epoch 3/10
156/156 [==============================] - 34s 219ms/step - loss: 0.2582 - categorical_accuracy: 0.9365 - val_loss: 3.8821 - val_categorical_accuracy: 0.0386
Epoch 4/10
156/156 [==============================] - 32s 203ms/step - loss: 0.1666 - categorical_accuracy: 0.9550 - val_loss: 3.9013 - val_categorical_accuracy: 0.0386
Epoch 5/10
156/156 [==============================] - 31s 201ms/step - loss: 0.1212 - categorical_accuracy: 0.9630 - val_loss: 4.2440 - val_categorical_accuracy: 0.0386
Epoch 6/10
156/156 [==============================] - 31s 201ms/step - loss: 0.0826 - categorical_accuracy: 0.9759 - val_loss: 4.2431 - val_categorical_accuracy: 0.0386
Epoch 7/10
156/156 [==============================] - 31s 198ms/step - loss: 0.0648 - categorical_accuracy: 0.9807 - val_loss: 4.3009 - val_categorical_accuracy: 0.0514
Epoch 8/10
156/156 [==============================] - 32s 205ms/step - loss: 0.0573 - categorical_accuracy: 0.9823 - val_loss: 4.3420 - val_categorical_accuracy: 0.0386
Epoch 9/10
156/156 [==============================] - 31s 196ms/step - loss: 0.0548 - categorical_accuracy: 0.9839 - val_loss: 4.4843 - val_categorical_accuracy: 0.0386
Epoch 10/10
156/156 [==============================] - 31s 200ms/step - loss: 0.0478 - categorical_accuracy: 0.9887 - val_loss: 4.7390 - val_categorical_accuracy: 0.0386

Run inference on training data and validation data:
156/156 [==============================] - 35s 227ms/step - loss: 4.7357 - categorical_accuracy: 0.0386
39/39 [==============================] - 11s 279ms/step - loss: 4.7326 - categorical_accuracy: 0.0386
Train Loss: 4.735676199961931; Train Acc: 0.03858520835638046
Val Loss: 4.732603843395527; Val Acc: 0.03858520835638046