Lightning-AI / Lightning-AI/lightning-thunder

Run `examine` automatically when hit unsupported operator failure

Open
#270 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

## 🚀 Feature

`examine` is a very useful and clean method that analyzes the network and reports the unsupported operators. When Thunder hits an unsupported operator, it would be very useful to have Thunder automatically run `examine` internally with the model and the inputs it already has and report the `examine` output along with the error.

### Motivation

As Thunder grows in adoption, we are seeing it being used on models with operators which are not currently supported. Today, Thunder throws an error that's not very clear that the error is attributed to an unsupported operator.
Example, see comment from a contributor below -- https://github.com/Lightning-AI/lightning-thunder/issues/93#issuecomment-2037641430

Another motivation for this comes from more complex libraries like NeMo where the model builder and the dataloader are not often in the same piece of code and is not easily accessible as they use PyTorch Lightning Trainer etc. While there is an option to create dummy inputs to run `examine`, this creates a layer of friction between adoption. Often, the engineer running the model is not the same as the one who built the model and may not even have the understanding of the input data expected by the model.

CC - @mruberry @tfogal

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing where Thunder reports unsupported-operator failures and how the existing `examine` entry point receives a model and its inputs. Done means those failures include the `examine` output generated from the model and inputs already available to Thunder.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
compilers
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.