Lightning-AI / Lightning-AI/lightning-thunder

Reentrant JIT for higher order operators

Open
#1,134 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

design enhancement interpreter jit
Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

## 🚀 Feature

Add support for PyTorch Callable -> Thunder Callable translation in Thunder JIT.

### Motivation

Several PyTorch operators accept a Python function with PyTorch operations inside as one of their arguments. In PyTorch, these operators are called "higher order operators". Examples of these operators:

- [torch.utils.checkpoint.checkpoint](https://pytorch.org/docs/stable/checkpoint.html#torch.utils.checkpoint.checkpoint)
- [torch.cond](https://pytorch.org/docs/stable/generated/torch.cond.html#torch.cond)
- [torch.associative_scan](https://github.com/pytorch/pytorch/blob/c18052da0eff728c96e29307381c90ee05951022/torch/_higher_order_ops/associative_scan.py#L83)
- [torch.scan](https://github.com/pytorch/pytorch/blob/c18052da0eff728c96e29307381c90ee05951022/torch/_higher_order_ops/scan.py#L46)
- [torch.while_loop](https://github.com/pytorch/pytorch/blob/c18052da0eff728c96e29307381c90ee05951022/torch/_higher_order_ops/while_loop.py#L62)

### Pitch

Thunder should support all of the above operators. It's easy to support only Thunder functions as inputs (example for `checkpoint` https://github.com/Lightning-AI/lightning-thunder/pull/1127), but the best user experience would be enabled by the automatic translation of user-provided PyTorch callables into Thunder ones while constructing the initial Thunder trace.

An example of `torch.cond` to support:
```py
import torch

def true_fn(x: torch.Tensor):
return torch.cos(x)
def false_fn(x: torch.Tensor):
return torch.sin(x)

# Ideally putting thunder.jit decorator should just work, this requires translation of true_fn and false_fn into Thunder functions so that the insides could be traced and understood by the rest of the system
# @thunder.jit
def f(true_fn, false_fn, x):
return torch.cond(x.shape[0] > 4, true_fn, false_fn, (x,))

x = torch.ones(5)
print(f(true_fn, false_fn, x))
```

### Alternatives

@t-vi, please fill in this section with details about alternative solutions.

Implement checkpointing via a lookaside that
- traces the checkpointed function as is,
- sets a flag in the JITCtx that effects the wrap callback to add a `rematerialize_for_backward` or so proxy tag to proxies that are wrapped (or created and wrapped),
- then clears the flag on the outputs.

The other higher order functions are prototypes currently, barring other pressing needs I think this should inform our prioritization. It would be a formidable change to the nature of traces to have higher order functions in them.
Before looking at this usecase, it would be good to figure out "call jitted function / module from jitted function" first, I guess this would be very useful for jitting training loops with optimizer steps.

### Additional context

An attempt at using jit inside lookasides currently fails: https://github.com/Lightning-AI/lightning-thunder/issues/1126.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading issue #1126 and pull request #1127, then inspect the JITCtx and wrap callback references in the issue. Compare the proposed checkpoint lookaside approach with the broader requirement for translating PyTorch callables during the initial Thunder trace. Done would require an agreed design and support for the listed higher-order operators, but the issue does not define tests or a concrete implementation boundary.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.