Lightning-AI / Lightning-AI/lightning-thunder

automatically add executor for "executor-linked" symbols

Open
#732 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

Idea: implement automatically adding the default executor for special-purpose symbols - i.e. the ability for symbols to request "if you see me and don't have any other executor for it, include executor foo".

Apparently, we often have transforms that introduce new symbols linked to one particular executor.

This means that the user has to apply the transform and specify the executor to be used in addition to the transform where it would seem to be evident that we need it for the new symbol.

So for e.g. quantization you currently need
```
jm = thunder.jit(
model_fp_reference,
executors=(bitsandbytes_executor,),
early_transforms=[BitsAndBytesLinearQuant4bit()],
)
```

but clearly, the `bnb_matmul_nf4` added to the graph by `BitsAndBytesLinearQuant4bit` belongs to the `bitsandbytes_executor`, so why would the user need to specify executors here (and decide whether they also want the default executors or not etc...).

This is from a discussion with @tfogal

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing thunder.jit's executor selection and the BitsAndBytesLinearQuant4bit transform, focusing on how the bnb_matmul_nf4 symbol is linked to bitsandbytes_executor. Define how requested executors should interact with explicitly supplied and default executors; done when the transform no longer requires manual executor specification and the behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
compilers, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.