quantized model only forward faster than float32, but if include get output, slower

Hi, I have a quantized model perfcount on time, quantized is faster:

def run_time(model_p):
    session = ort.InferenceSession(model_p)
    input_name = session.get_inputs()[0].name
    total = 0.0
    runs = 10
    input_data = np.zeros((1, 3, 224, 224), np.float32)
    _ = session.run([], {input_name: input_data})
    for i in range(runs):
        start = time.perf_counter()
        _ = session.run([], {input_name: input_data})
        end = (time.perf_counter() - start) * 1000
        total += end
        print(f"{end:.2f}ms")
    total /= runs
    print(f"Avg: {total:.2f}ms")

Output:

7.57ms
7.45ms
7.44ms
7.37ms
7.42ms
7.48ms
7.65ms
7.46ms
7.39ms
7.39ms
Avg: 7.46ms
5.01ms
5.27ms
5.06ms
5.01ms
5.00ms
4.98ms
5.03ms
4.99ms
5.00ms
5.05ms
Avg: 5.04ms

int8 faster.

But, when eval it, get output and calculate max value, it become slower:

def evaluate_onnx_model(model_p, test_loader, criterion=None):
    running_loss = 0
    running_corrects = 0

    session = ort.InferenceSession(model_p)
    input_name = session.get_inputs()[0].name

    total = 0.
    for inputs, labels in test_loader:
        inputs = inputs.cpu().numpy()
        labels = labels.cpu().numpy()

        start = time.perf_counter()
        outputs = session.run([], {input_name: inputs})[0]
        end = (time.perf_counter() - start)
        total += end

        preds = np.argmax(outputs, 1)
        if criterion is not None:
            loss = criterion(outputs, labels).item()
        else:
            loss = 0
        # statistics
        running_corrects += np.sum(preds == labels)

    # eval_loss = running_loss / len(test_loader.dataset)
    eval_accuracy = running_corrects / len(test_loader.dataset)
    total /= len(test_loader)
    print(f"eval loss: {0}, eval acc: {eval_accuracy}, cost: {total}")
    return 0, eval_accuracy

eval loss: 0, eval acc: 0.8477, cost: 0.9931462904438376
eval loss: 0, eval acc: 0.8345, cost: 1.501858500018716

the cost is slower than foat32 model.... How could this be?

Start by reproducing the timing in run_time and evaluate_onnx_model, comparing session.run alone with the version that retrieves outputs and computes predictions. Record the model files, ONNX Runtime version, execution provider, hardware, batch sizes, and output shapes, then profile both paths. Done means providing a reproducible explanation for the timing difference or identifying the missing environmental detail.