pytorch / pytorch/benchmark

[eager_fail_to_run] opacus_cifar10

Open
#2,568 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
346
PR merge metrics
No merged PRs in 30d

Description

Running from pytorch main

python benchmarks/dynamo/torchbench.py --only opacus_cifar10 --accuracy --no-translation-validation --training --amp --backend inductor --device cuda --output out.csv
cuda train opacus_cifar10
Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2744, in validate_model
    self.model_iter_fn(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 455, in forward_and_backward_pass
    self.grad_scaler.scale(loss).backward()
  File "/data/users/ivankobzarev/a/pytorch/torch/_tensor.py", line 648, in backward
    torch.autograd.backward(
  File "/data/users/ivankobzarev/a/pytorch/torch/autograd/__init__.py", line 353, in backward
    _engine_run_backward(
  File "/data/users/ivankobzarev/a/pytorch/torch/autograd/graph.py", line 815, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 86, in __call__
    return self.hook(module, *args, **kwargs)
  File "/data/users/ivankobzarev/a/opacus/opacus/grad_sample/grad_sample_module.py", line 327, in capture_backprops_hook
    activations, backprops = self.rearrange_grad_samples(
  File "/data/users/ivankobzarev/a/opacus/opacus/grad_sample/grad_sample_module.py", line 383, in rearrange_grad_samples
    raise ValueError(
ValueError: No activations detected for <class 'torch.nn.modules.linear.Linear'>, run forward after add_hooks(model)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 4857, in run
    ) = runner.load_model(
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 372, in load_model
    self.validate_model(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2746, in validate_model
    raise RuntimeError("Eager run failed") from e
RuntimeError: Eager run failed

eager_fail_to_run

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the listed benchmarks/dynamo/torchbench.py command, then start at forward_and_backward_pass and validate_model in the referenced files. Investigate the opacus_cifar10 eager training traceback; done when the benchmark completes its eager training run without the reported missing-activations error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.