MobileNetV3 quantization bug

🐛 Describe the bug

This issue documents an bug discovered on February 2021 while quantizing the MobileNetV3. We worked around the issue but it probably remains unresolved on the eager quantization API. Here I'm gathering all the information exchanged with the quantization team so that we can investigate on the future.

TLDR: During the QAT training of MobileNetV3 the accuracies are at the expected levels. After serializing and loading the weights of the quantized weights, the accuracies appear similar to a randomly initialized model. The workaround is to instead load the QAT fine-tuned weights and convert the model.

Originally to quantize the MobileNetV3 Large we took the standard approach (see https://github.com/pytorch/vision/pull/3323). Our initial setup was extremely similar to the one of MobileNetV2. We trained a model with the reference script and we got good accuracies. Unfortunately when we tried to deploy the weights of the quantized model we found out that loading them and doing predictions leads to extremely low accuracy (less than 1%).

After trial and error, we found implemented the following workaround that allows to load the weights, quantize the model and get good accuracies: https://github.com/pytorch/vision/pull/3323/commits/274c6a1393384054876d701ffa1b54eb6750f1d8. Effectively we create a QAT model, load the fine-tuned weights and then quantize it.

The above workaround should be unnecessary. Instead MobileNetV3 should follow the same approach as with every other QAT model (such as in the case of MobileNetV2).

Start with the MobileNetV3 quantization setup described in the reference script and compare it with MobileNetV2's QAT flow. Review torchvision PR 3323 and commit 274c6a1393384054876d701ffa1b54eb6750f1d8, then investigate the eager quantization API behavior around serialization and loading. Done means MobileNetV3 can load quantized weights and retain expected accuracy without the documented workaround.