patrick-kidger / patrick-kidger/diffrax
Question about an “AllSaveAdjoint” (store-all-steps, no recomputation) variant
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 189
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 1
Description
Thanks for the amazing project. I find it very useful and blazing fast in quantum simulation, but do not know the implementation details about the solvers. Here is how I trying to understand how these solver working
Currently Diffrax exposes several adjoint/backpropagation strategies: RecursiveCheckpointAdjoint (default, binomial checkpointing), DirectAdjoint (forward+reverse capable but less efficient), and BacksolveAdjoint (continuous adjoint; not recommended due to approximate gradients). I did not find an option that simply stores every intermediate solver state once and performs a single reverse sweep with zero recomputation (time O(n), memory O(n)); the closest seems to be increasing the number of checkpoints in RecursiveCheckpointAdjoint.
Questions / motivation:
- Is there a deliberate design reason for not exposing a convenience variant that forces saving every step (e.g.
AllSaveAdjoint), given that users with ample memory might prefer pure O(n) time without the logn factor? - Can setting checkpoints to a very large number (e.g. equal to or exceeding actual step count) already emulate “all-save”, or are there structural evictions / internal policies that still cause recomputation even when memory allows?
Use cases:
- Small-to-moderate problems where n is not huge (such as quantum simulation of small quantum systems, this is basically all we can do in classic computer, which means this may be the full use case), GPU memory is plentiful, and wall-clock latency per training step matters more than conserving RAM.
- Benchmarking: having a no-recompute baseline helps quantify the overhead of checkpoint heuristics or of
DirectAdjoint. - Situations combining reverse-mode gradients with experimentation on solver internals where deterministic “no replay” profiling is simpler.
I'm new to differential equation solvers, if there are any misunderstandings please let me know, I'd appreciate it :)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the documented RecursiveCheckpointAdjoint, DirectAdjoint, and BacksolveAdjoint entry points and the surrounding discussion. Determine whether a large checkpoint count already provides the requested no-recomputation behavior, and document the design decision or requirements for an all-save variant.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100