Lightning-AI / Lightning-AI/lightning-thunder
AssertionError from process_recorded_modifications
@t-vi is already working on this.
Since Nov 11, 2024.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
## 🐛 Bug
I get the following assertion error from Thunder JIT:
```py
File ~/dev/lightning-thunder/thunder/core/jit_ext.py:1731, in thunder_general_jit(fn, args, kwargs, record_history, sharp_edges, ad_hoc_executor)
1729 prims.python_return(result)
1730 computation_trace.set_current_source_location(None, None)
-> 1731 process_recorded_modifications(ctx, epilogue_trace)
1732 last_interpreter_log = jfn._last_interpreter_log
1734 pro_to_comp, computation_intermediates = get_computation_inputs_and_intermediates(computation_trace)
File ~/dev/lightning-thunder/thunder/core/jit_ext.py:1610, in process_recorded_modifications(ctx, epilogue_trace)
1608 if inst == PseudoInst.STORE_SUBSCR:
1609 (value,) = args
-> 1610 assert isinstance(value.value, Proxy)
1612 assert modified_object.provenance.inst is PseudoInst.LOAD_ATTR
1613 assert modified_object.provenance.inputs[1].inst is PseudoInst.CONSTANT
AssertionError:
```
using the following script:
```py
import torch
import thunder
from transformers import Qwen2Config, Qwen2ForCausalLM
# https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/blob/main/config.json
configuration = Qwen2Config(
# Qwen2.5-7B-Instruct uses Grouped-Query Attention, while the default
# config uses Multi-Head Attention
num_attention_heads=28,
num_key_value_heads=4,
# Scaled down for testing
hidden_size=56,
vocab_size=4096,
)
configuration.num_hidden_layers = 1
with torch.device("cuda"):
model = Qwen2ForCausalLM(configuration).to(torch.bfloat16)
compiled_model = thunder.jit(model)
input_ids = torch.randint(0, configuration.vocab_size, (1, configuration.max_position_embeddings), device="cuda")
compiled_output = compiled_model(input_ids=input_ids, labels=input_ids)
```
The `transformers` version is 4.45.2
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.