[eager_fail_to_run] opacus_cifar10
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 346
- PR merge metrics
- No merged PRs in 30d
Description
Running from pytorch main
python benchmarks/dynamo/torchbench.py --only opacus_cifar10 --accuracy --no-translation-validation --training --amp --backend inductor --device cuda --output out.csv
cuda train opacus_cifar10
Traceback (most recent call last):
File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2744, in validate_model
self.model_iter_fn(model, example_inputs)
File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 455, in forward_and_backward_pass
self.grad_scaler.scale(loss).backward()
File "/data/users/ivankobzarev/a/pytorch/torch/_tensor.py", line 648, in backward
torch.autograd.backward(
File "/data/users/ivankobzarev/a/pytorch/torch/autograd/__init__.py", line 353, in backward
_engine_run_backward(
File "/data/users/ivankobzarev/a/pytorch/torch/autograd/graph.py", line 815, in _engine_run_backward
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 86, in __call__
return self.hook(module, *args, **kwargs)
File "/data/users/ivankobzarev/a/opacus/opacus/grad_sample/grad_sample_module.py", line 327, in capture_backprops_hook
activations, backprops = self.rearrange_grad_samples(
File "/data/users/ivankobzarev/a/opacus/opacus/grad_sample/grad_sample_module.py", line 383, in rearrange_grad_samples
raise ValueError(
ValueError: No activations detected for <class 'torch.nn.modules.linear.Linear'>, run forward after add_hooks(model)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 4857, in run
) = runner.load_model(
File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 372, in load_model
self.validate_model(model, example_inputs)
File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2746, in validate_model
raise RuntimeError("Eager run failed") from e
RuntimeError: Eager run failed
eager_fail_to_run
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the listed benchmarks/dynamo/torchbench.py command, then start at forward_and_backward_pass and validate_model in the referenced files. Investigate the opacus_cifar10 eager training traceback; done when the benchmark completes its eager training run without the reported missing-activations error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100