Accuracy drop : DeiT-tiny & LLaMA behavior differences between FP / Decomposition / FQ / TOSA

🐛 Describe the bug

Hi Executorch ARM backend team,

I’m validating the lowering pipeline and collected the following comparison across several models. Quantize with 8-bit only.
Decompose with aten: Decompose by arm backend quantizer.transform_for_annotation

With the experiments, the variables are as following.

• Executorch commit: 913436a4
• TOSA results are inferencing by tosa_reference_model
• For all models, test input is torch.rand(input_shape) (batch size:1)
• For verification only, calibration data == testing input.
• The models are using executorch/example/models
• LLaMA weights:

torch.manual_seed(0)
for p in self.model_.parameters():
# p.data.fill_(0)
torch.nn.init.normal_(p, mean=0.0, std=0.02)
for b in self.model_.buffers():
b.data.fill_(0)

Based on the conditions above, I have some questions about the comparison results:

1. I can roughly understand why DeiT-tiny shows a large difference between FP and its quantized output — this may come from limited quantization granularity or the nature of the model itself. In that sense, it’s reasonable that the TOSA output would also deviate significantly from FP.
   However, what I’m not fully sure about is FQ vs TOSA. In theory, these two should be very close to each other, similar to the other models.
2. For LLaMA, there is already a very large deviation from the FP model immediately after applying quantizer.transform_for_annotation, even before running FakeQuant or lowering to TOSA.

Is it possible that these models are currently lowerable, but numerical correctness is not yet guaranteed for them?

cc @freddan80 @per @zingo @oscarandersson8218 @digantdesai

Start by reproducing the reported comparisons from ExecuTorch commit 913436a4 using the models in executorch/example/models and the stated random inputs and calibration data. Trace the ARM quantization path at quantizer.transform_for_annotation, then compare FakeQuant and tosa_reference_model outputs for DeiT-tiny and LLaMA. Done means the source of the discrepancies and the numerical-correctness expectations for these models are established.