nan gradients

not sure what the etiology is here, but figured we should document it and maybe centralize repair discussion here since it's apparently a known issue. I think with CARPFilip we first observed this when gradient checkpointing was turned off, prior to which biases had feasible gradient distributions but weights gradients were already reporting as collapsed to zero (or more likely, had exploded and the nan's were interpreted as zero by wandb or the gradient checkpointing mechanism).

No files, tests, or entry points are identified. Start by reproducing the gradient behavior with gradient checkpointing enabled and disabled, then inspect how zero or NaN gradients are reported; done means the cause and any repair path are established and documented in this issue.

Written by the indexing model from the issue text.