Lightning-AI / Lightning-AI/lightning-thunder

AssertionError from process_recorded_modifications

Open
#1,405 5 comments 0 reactions 1 assignee View on GitHub

@t-vi is already working on this.

Since Nov 11, 2024.

huggingface jit
Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

## 🐛 Bug
I get the following assertion error from Thunder JIT:
```py
File ~/dev/lightning-thunder/thunder/core/jit_ext.py:1731, in thunder_general_jit(fn, args, kwargs, record_history, sharp_edges, ad_hoc_executor)
1729 prims.python_return(result)
1730 computation_trace.set_current_source_location(None, None)
-> 1731 process_recorded_modifications(ctx, epilogue_trace)
1732 last_interpreter_log = jfn._last_interpreter_log
1734 pro_to_comp, computation_intermediates = get_computation_inputs_and_intermediates(computation_trace)

File ~/dev/lightning-thunder/thunder/core/jit_ext.py:1610, in process_recorded_modifications(ctx, epilogue_trace)
1608 if inst == PseudoInst.STORE_SUBSCR:
1609 (value,) = args
-> 1610 assert isinstance(value.value, Proxy)
1612 assert modified_object.provenance.inst is PseudoInst.LOAD_ATTR
1613 assert modified_object.provenance.inputs[1].inst is PseudoInst.CONSTANT

AssertionError:
```
using the following script:
```py
import torch
import thunder
from transformers import Qwen2Config, Qwen2ForCausalLM

# https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/blob/main/config.json
configuration = Qwen2Config(
# Qwen2.5-7B-Instruct uses Grouped-Query Attention, while the default
# config uses Multi-Head Attention
num_attention_heads=28,
num_key_value_heads=4,
# Scaled down for testing
hidden_size=56,
vocab_size=4096,
)
configuration.num_hidden_layers = 1
with torch.device("cuda"):
model = Qwen2ForCausalLM(configuration).to(torch.bfloat16)

compiled_model = thunder.jit(model)
input_ids = torch.randint(0, configuration.vocab_size, (1, configuration.max_position_embeddings), device="cuda")
compiled_output = compiled_model(input_ids=input_ids, labels=input_ids)
```

The `transformers` version is 4.45.2

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.