tinygrad / tinygrad/tinygrad

gradients through where when one of the branches is nan

Open
#12,409 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
33.6k
Forks
4.3k
Avg merge
3h 11m
Merged PRs (30d)
491

Description

What is tinygrad's plan to deal with gradients of functions defined with a Tensor.where operation in which the branch not taken is nan but the gradient still comes out to nan due to 0*nan resulting in nan. The issue is present in jax as well and is described in detail here - https://github.com/jax-ml/jax/issues/1052#issuecomment-514083352.

It can be reproduced in tinygrad with something like

from tinygrad import Tensor

x = Tensor([0], dtype="float32", requires_grad=True)
y = Tensor.where(x == 0, 0, x * x.sqrt())
y.sum().backward()
print(x.grad.tolist())

As they mention, the right solution which "is pretty heavyweight", "is something like tracking a symbolic zero mask potentially through the entire backward pass of differentiation".

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided Tensor.where reproduction and compare its gradient behavior with the linked JAX discussion. Trace the backward pass for Tensor.where and determine the intended handling of a NaN in the unselected branch; done should include a documented plan and a verified result for this example.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.