Lightning-AI / Lightning-AI/lightning-thunder
Reentrant JIT for higher order operators
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
## 🚀 Feature
Add support for PyTorch Callable -> Thunder Callable translation in Thunder JIT.
### Motivation
Several PyTorch operators accept a Python function with PyTorch operations inside as one of their arguments. In PyTorch, these operators are called "higher order operators". Examples of these operators:
- [torch.utils.checkpoint.checkpoint](https://pytorch.org/docs/stable/checkpoint.html#torch.utils.checkpoint.checkpoint)
- [torch.cond](https://pytorch.org/docs/stable/generated/torch.cond.html#torch.cond)
- [torch.associative_scan](https://github.com/pytorch/pytorch/blob/c18052da0eff728c96e29307381c90ee05951022/torch/_higher_order_ops/associative_scan.py#L83)
- [torch.scan](https://github.com/pytorch/pytorch/blob/c18052da0eff728c96e29307381c90ee05951022/torch/_higher_order_ops/scan.py#L46)
- [torch.while_loop](https://github.com/pytorch/pytorch/blob/c18052da0eff728c96e29307381c90ee05951022/torch/_higher_order_ops/while_loop.py#L62)
### Pitch
Thunder should support all of the above operators. It's easy to support only Thunder functions as inputs (example for `checkpoint` https://github.com/Lightning-AI/lightning-thunder/pull/1127), but the best user experience would be enabled by the automatic translation of user-provided PyTorch callables into Thunder ones while constructing the initial Thunder trace.
An example of `torch.cond` to support:
```py
import torch
def true_fn(x: torch.Tensor):
return torch.cos(x)
def false_fn(x: torch.Tensor):
return torch.sin(x)
# Ideally putting thunder.jit decorator should just work, this requires translation of true_fn and false_fn into Thunder functions so that the insides could be traced and understood by the rest of the system
# @thunder.jit
def f(true_fn, false_fn, x):
return torch.cond(x.shape[0] > 4, true_fn, false_fn, (x,))
x = torch.ones(5)
print(f(true_fn, false_fn, x))
```
### Alternatives
@t-vi, please fill in this section with details about alternative solutions.
Implement checkpointing via a lookaside that
- traces the checkpointed function as is,
- sets a flag in the JITCtx that effects the wrap callback to add a `rematerialize_for_backward` or so proxy tag to proxies that are wrapped (or created and wrapped),
- then clears the flag on the outputs.
The other higher order functions are prototypes currently, barring other pressing needs I think this should inform our prioritization. It would be a formidable change to the nature of traces to have higher order functions in them.
Before looking at this usecase, it would be good to figure out "call jitted function / module from jitted function" first, I guess this would be very useful for jitting training loops with optimizer steps.
### Additional context
An attempt at using jit inside lookasides currently fails: https://github.com/Lightning-AI/lightning-thunder/issues/1126.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading issue #1126 and pull request #1127, then inspect the JITCtx and wrap callback references in the issue. Compare the proposed checkpoint lookaside approach with the broader requirement for translating PyTorch callables during the initial Thunder trace. Done would require an agreed design and support for the listed higher-order operators, but the issue does not define tests or a concrete implementation boundary.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- compilers
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100