gradients through where when one of the branches is nan
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 33.6k
- Forks
- 4.3k
- Avg merge
- 3h 11m
- Merged PRs (30d)
- 491
Description
What is tinygrad's plan to deal with gradients of functions defined with a Tensor.where operation in which the branch not taken is nan but the gradient still comes out to nan due to 0*nan resulting in nan. The issue is present in jax as well and is described in detail here - https://github.com/jax-ml/jax/issues/1052#issuecomment-514083352.
It can be reproduced in tinygrad with something like
from tinygrad import Tensor
x = Tensor([0], dtype="float32", requires_grad=True)
y = Tensor.where(x == 0, 0, x * x.sqrt())
y.sum().backward()
print(x.grad.tolist())
As they mention, the right solution which "is pretty heavyweight", "is something like tracking a symbolic zero mask potentially through the entire backward pass of differentiation".
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided Tensor.where reproduction and compare its gradient behavior with the linked JAX discussion. Trace the backward pass for Tensor.where and determine the intended handling of a NaN in the unselected branch; done should include a documented plan and a verified result for this example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100