[eager_fail_to_run] cuda train sam

python benchmarks/dynamo/torchbench.py --only sam --accuracy --no-translation-validation --training --amp --backend inductor --disable-cudagraphs --device cuda

cuda train sam
Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2744, in validate_model
    self.model_iter_fn(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 455, in forward_and_backward_pass
    self.grad_scaler.scale(loss).backward()
  File "/home/ivankobzarev/local/a/pytorch-env/lib/python3.10/site-packages/torch/_tensor.py", line 581, in backward
    torch.autograd.backward(
  File "/home/ivankobzarev/local/a/pytorch-env/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "/home/ivankobzarev/local/a/pytorch-env/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 4857, in run
    ) = runner.load_model(
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 372, in load_model
    self.validate_model(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2746, in validate_model
    raise RuntimeError("Eager run failed") from e
RuntimeError: Eager run failed

eager_fail_to_run

Start by rerunning the provided torchbench.py command for the SAM model with CUDA training enabled. Inspect benchmarks/dynamo/torchbench.py at forward_and_backward_pass and benchmarks/dynamo/common.py at validate_model to trace the eager failure. Done means the SAM training benchmark completes its eager validation without the reported backward error.