[QUESTION] How to debug missing gradients?
@eric-heiden is already working on this.
Since Apr 9, 2025.
- Dominant language
- Python
- Stars
- 7.1k
- Forks
- 624
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 5
Description
I run simulations with various meshes. Sometimes, they can take on unconventional shapes. I would like to compute gradients of some simple loss functions, e.g., the distance traveled under gravity impact in a certain time.
Usually, this works just fine (using the "hack" described in #610 to get the gradients in the first place). However, in some (unpredictable) cases, the gradients are all-zero. I have not yet found a way to reproduce this (unfortunately it is not feasible to extract standalone code to reproduce just this issue from my codebase right now), but based on my experience, this happens
- When the meshes become too "edgy" / asymetric
- faster, when I use finer meshes (rule of thumb: at least 1200 vertices).
To demonstrate what I mean, I attached a small video of a simulation for which the gradient computation fails. If I change minor details about the mesh, I get nonzero gradients for all contact points again. Clearly, there is contact, and that contact must result in some non-zero impact on the final outcome. Still, the gradients are zero.
My first guess was that there are undetected nan or inf values somewhere in the computation of the gradients - but going through all computations step by step, I can't find anything that's wrong. The visualization of the tape also looks good (and does not change between "good" and "dysfunctional" simulations).
I have no idea how to detect what's going wrong here anymore - if there's another debugging tool or a hunch of where I should look, I'd appreciate it!
https://github.com/user-attachments/assets/470a4610-df46-44e5-bb94-312865040a5f
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.