NVIDIA / NVIDIA/cuEquivariance

`AttentionPairBiasMask.backward()` fails with non-contiguous gradient tensors despite contiguous forward inputs

Open
#177 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
433
Forks
42
PR merge metrics
No merged PRs in 30d

Description

We're encountering a runtime error in AttentionPairBiasMask.backward() during training:

RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

The error occurs at line 358 in attention_pair_bias_torch.py:

grad_out_mask = grad_out_mask.view(B, ctx.num_heads, -1)

All input tensors to cuet.attention_pair_bias() in the forward pass are verified contiguous, and the forward pass completes successfully every time. The error occurs during backpropagation when gradient tensors become non-contiguous.

Environment

  • Hardware: H100 GPUs
  • cuEquivariance version: 0.6.1
  • PyTorch version: 2.7
  • CUDA version: 12.x

We've implemented a monkey patch that makes gradient tensors contiguous before calling the original backward function, which resolves the issue. However:

  • Should AttentionPairBiasMask.backward() handle non-contiguous gradient tensors? (A simple fix would be changing view() to reshape())
  • Or is it expected that all gradient inputs will be contiguous? If so, what's the recommended approach for ensuring this?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in attention_pair_bias_torch.py at line 358 and inspect AttentionPairBiasMask.backward() with the reported non-contiguous gradient. Reproduce the H100/PyTorch 2.7 failure if possible, then verify that backward completes for non-contiguous gradients without changing the contiguous-forward behavior; the issue identifies reshape as a candidate fix but does not name a test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.