NVIDIA / NVIDIA/warp

[QUESTION] How to debug missing gradients?

Open
#640 4 comments 0 reactions 2 assignees View on GitHub

@eric-heiden is already working on this.

Since Apr 9, 2025.

question
Dominant language
Python
Stars
7.1k
Forks
624
Avg merge
3d 17h
Merged PRs (30d)
5

Description

I run simulations with various meshes. Sometimes, they can take on unconventional shapes. I would like to compute gradients of some simple loss functions, e.g., the distance traveled under gravity impact in a certain time.

Usually, this works just fine (using the "hack" described in #610 to get the gradients in the first place). However, in some (unpredictable) cases, the gradients are all-zero. I have not yet found a way to reproduce this (unfortunately it is not feasible to extract standalone code to reproduce just this issue from my codebase right now), but based on my experience, this happens

  • When the meshes become too "edgy" / asymetric
  • faster, when I use finer meshes (rule of thumb: at least 1200 vertices).

To demonstrate what I mean, I attached a small video of a simulation for which the gradient computation fails. If I change minor details about the mesh, I get nonzero gradients for all contact points again. Clearly, there is contact, and that contact must result in some non-zero impact on the final outcome. Still, the gradients are zero.

My first guess was that there are undetected nan or inf values somewhere in the computation of the gradients - but going through all computations step by step, I can't find anything that's wrong. The visualization of the tape also looks good (and does not change between "good" and "dysfunctional" simulations).

I have no idea how to detect what's going wrong here anymore - if there's another debugging tool or a hunch of where I should look, I'd appreciate it!

https://github.com/user-attachments/assets/470a4610-df46-44e5-bb94-312865040a5f

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.