Gradient `tape` vs. `adjoint=True`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.1k
- Forks
- 624
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 5
Description
Are there any guidelines, when to use the wp.tape or the adjoint=True argument to compute gradients?
There are two examples of a torch.autograd.Function in this repository using different approches.
https://github.com/NVIDIA/warp/blob/79a56a9c2754bbec1418e490efafd43091056856/examples/example_sim_fk_grad_torch.py#L29
https://github.com/NVIDIA/warp/blob/79a56a9c2754bbec1418e490efafd43091056856/warp/tests/test_torch.py#L416
I tried to expand the test_torch.py with multiple inputs, but I wasn't able to get it to work (reliably!). Usually I get Warp CUDA error 1: invalid argument (/buildAgent/work/a9ae500d09a78409/warp/native/warp.cu:1891) but sometimes the program segfaults instead.
Is there anything I would need to be aware of when using the adjoint=True argument?
Today my college experimented with the wp.tape approach, and we converged on the code pasted below.
We had to add the ctx.x0.grad.zero_() line to make sure that multiple calls to pytorch's backward() are working properly (to please gradcheck())
But even more importantly we had to add .clone(), requires_grad=True) to ensure that warp does not write into the pytroch's gradient buffers directly, since that results in gradients to be 2x of the true gradient if the variable is a leaf in the computation graph!
-> We haven't tested it, but that would suggest there is a bug in the the example_sim_fk_grad_torch example, no?
import numpy as np
import torch
from torch.autograd import gradcheck
import warp as wp
wp.init()
device = "cuda"
torch_device = wp.device_to_torch(device)
wp_device = wp.device_from_torch(torch_device)
@wp.kernel()
def op_kernel(
x0: wp.array(dtype=wp.float32),
x1: wp.array(dtype=wp.float32),
x2: wp.array(dtype=wp.float32),
y: wp.array(dtype=wp.float32),
):
i = wp.tid()
y[i] = x0[i] ** 2.0 * x1[i] ** 2.0 * x2[i] ** 2.0
class WPGradTape(torch.autograd.Function):
@staticmethod
def forward(
ctx,
x0,
x1,
x2,
):
wp.synchronize_device()
ctx.tape = wp.Tape()
ctx.x0 = wp.from_torch(x0.clone(), requires_grad=True)
ctx.x1 = wp.from_torch(x1.clone(), requires_grad=True)
ctx.x2 = wp.from_torch(x2.clone(), requires_grad=True)
ctx.y = wp.empty(x1.shape[0], dtype=wp.float32, device=wp_device)
with ctx.tape:
wp.launch(
kernel=op_kernel,
dim=[len(x1)],
inputs=[
ctx.x0,
ctx.x1,
ctx.x2,
],
outputs=[ctx.y],
adjoint=False,
device=wp_device,
)
wp.synchronize_device()
return wp.to_torch(ctx.y)
@staticmethod
def backward(ctx, adj_y):
wp.synchronize_device()
ctx.x0.grad.zero_()
ctx.x1.grad.zero_()
ctx.x2.grad.zero_()
ctx.y.grad = wp.from_torch(adj_y).contiguous()
ctx.tape.backward()
wp.synchronize_device()
return (
wp.to_torch(ctx.tape.gradients[ctx.x0]),
wp.to_torch(ctx.tape.gradients[ctx.x1]),
wp.to_torch(ctx.tape.gradients[ctx.x2]),
)
torch.manual_seed(42)
n = 3
# input data
x0_base = torch.rand(n, dtype=torch.float32, device=torch_device, requires_grad=True)
x1_base = torch.rand(n, dtype=torch.float32, device=torch_device, requires_grad=True)
x2_base = torch.rand(n, dtype=torch.float32, device=torch_device, requires_grad=True)
x0 = x0_base * 1.0
x1 = x1_base * 1.0
x2 = x2_base * 1.0
def fun(x0, x1, x2):
return x0**2 * x1**2 * x2**2
gradcheck(
WPGradTape.apply,
# fun,
(
x0_base,
x1_base,
x2_base,
),
atol=1e-1,
rtol=1e-1,
)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Compare examples/example_sim_fk_grad_torch.py with warp/tests/test_torch.py, focusing on the torch.autograd.Function implementations and their use of wp.Tape and adjoint=True. Reproduce the reported multi-input gradcheck failure and CUDA or segmentation errors. Done means the gradient behavior is reliable and the appropriate usage guidance or example corrections are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100