pytorch / pytorch/benchmark

[eager_fail_to_run] resnet50_quantized_qat

Open
#2,569 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
346
PR merge metrics
No merged PRs in 30d

Description

Running from pytorch main

python benchmarks/dynamo/torchbench.py --only resnet50_quantized_qat --accuracy --no-translation-validation --training --amp --backend inductor --disable-cudagraphs --device cuda --output out.csv
cuda train resnet50_quantized_qat
Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2744, in validate_model
    self.model_iter_fn(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 453, in forward_and_backward_pass
    pred = mod(*cloned_inputs)
  File "/data/users/ivankobzarev/a/pytorch/torch/fx/graph_module.py", line 824, in call_wrapped
    return self._wrapped_call(self, *args, **kwargs)
  File "/data/users/ivankobzarev/a/pytorch/torch/fx/graph_module.py", line 400, in __call__
    raise e
  File "/data/users/ivankobzarev/a/pytorch/torch/fx/graph_module.py", line 387, in __call__
    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 1749, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 1760, in _call_impl
    return forward_call(*args, **kwargs)
  File "<eval_with_key>.3", line 167, in forward
    activation_post_process_73 = self.activation_post_process_73(fc);  fc = None
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 1749, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 1760, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/users/ivankobzarev/a/pytorch/torch/ao/quantization/fake_quantize.py", line 408, in forward
    return torch.fused_moving_avg_obs_fake_quant(
RuntimeError: expected scalar type Float but found Half

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 4857, in run
    ) = runner.load_model(
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 372, in load_model
    self.validate_model(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2746, in validate_model
    raise RuntimeError("Eager run failed") from e
RuntimeError: Eager run failed

eager_fail_to_run

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the provided benchmarks/dynamo/torchbench.py command and inspect forward_and_backward_pass in that file. Then trace the Float/Half mismatch through torch/ao/quantization/fake_quantize.py; done means resnet50_quantized_qat completes eager training with the command without the scalar-type error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.