[DeepLab] How to use my own tf_initial_checkpoint, specifically the resulting graph from a model trained on deeplab the usual way on another dataset?

In DeepLab v3+, I have managed to my use own dataset and train on it using a tf_initial_checkpoint from pascalvoc pre-trained models provided at the Model Zoo. These were some parameters as set during training.

--initialize_last_layer=False
--last_layers_contain_logits_only=True
--fine_tune_batch_norm=True \

Now I have the resulting model checkpoints and a frozen graph export (.pb) from this training.

I have 6 other datasets which are subsets of this previous dataset, and I now wish to train on them separately (they have different number of classes). For that, I would want to use the initial checkpoint as this previously trained model.

Upon trying to do that (setting tf_initial_checkpoint to .../xyzdir/model.ckpt , where .../xyzdir/ contains copies of model checkpoints and frozen_inference_graph.pb from previous training), deeplab does acknowledge the presence of this checkpoint, but starts training from zero nevertheless. MioUs obtained from using the frozen graph directly on my new dataset are ~0.6, and using this newly trained one will start from absolute zero learning and absymal MioUs.

How to use my checkpoints properly as initial checkpoints for other datasets? Does this even make sense, since only the last layer might be getting re-initialized resulting in training from scratch.

I should mention that the exact names of produced .pb and .ckpt files were different, I tried using them as is, and also tried after renaming them to the generic frozen_inference_graph.pb and model.ckpt.index )