Calibration ranges for Histogram-based approaches gives very bad performances on the task (BERT / SQuAD).

Hi ORT folks 👋🏻.

I continue my walk over all the possible calibration techniques provided by ORT, especially target BERT-like models, and spot something strange w.r.t the calibration ranges generated by different Calibrater.

I'm using the following schema:

• activation: uint8, asymmetric
• weight: int8, symmetric
• reduce_range: false
• quant_format: "qdq"

Everything is highly inspired by the ORT example on BERT-SQuAD

Every attempt with a calibration method different than minmax leads to very poor exact match and F1 score (like EM < 0.01 and F1 < 0.1)

The baseline I'm using is from HuggingFace's Hub: https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad

• full-precision model: EM = 85.875, F1 = 92.080
• u8/s8, minmax, 256 samples: EM = 84.588, F1 = 91.519
• u8/s8, entropy, 256 samples, 128 bins: EM = 82.980, F1 = 90.468
• u8/s8, percentiles, 256 samples, 2048 bins, 99.999%: EM = 0.302 , F1 = 5.82

when looking at the calibration ranges for percentiles I'm confused by the results (see JSON below).

Am I missing something? I would have expected to have a negative side on each node (as I get with entropy).

Also, histogram-based values are symmetric, the left-side values are always the negative of the right-side. Is it expected? Even in asymmetric mode?

Thanks a lot for your inputs on this 🙏🏻

{
  "916": [
    6.055913925170898,
    6.055913925170898
  ],
  "367": [
    1.7712894678115845,
    1.7712894678115845
  ],
  "438": [
    6.504486560821533,
    6.504486560821533
  ],
  "356": [
    3.519847869873047,
    3.519847869873047
  ],
  "898": [
    9.999778782798785e-13,
    9.999778782798785e-13
  ],
  "226": [
    43.03803253173828,
    43.03803253173828
  ],
  "127": [
    0.011049754917621613,
    0.011049754917621613
  ],