Lightning-AI / Lightning-AI/lightning-thunder
[NeVa] [rank0]: TypeError: matmul(): argument 'input' (position 1) must be Tensor, not TensorProxy
@tfogal is already working on this.
Since Dec 9, 2024.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
While preparing the benchmark for eager and dynamo using the code from the fork: https://github.com/tfogal/NeMo I get errors for dynamo case.
## 🐛 Bug
After fixing [1187](https://github.com/Lightning-AI/lightning-thunder/issues/1187) NeMo NeVa dynamo throws a new error
```python
model.model = torch.compile(backend=thunder_backend, dynamic=False)(model.model)
```
it throws:
```
[rank0]: File "/usr/local/lib/python3.10/dist-packages/torch/fx/graph_module.py", line 359, in __call__
[rank0]: raise e.with_traceback(None) # noqa: B904
[rank0]: TypeError: matmul(): argument 'input' (position 1) must be Tensor, not TensorProxy
```
### To Reproduce
Steps to reproduce the behavior:
1. Clone: `https://github.com/tfogal/NeMo`
2. Use latest lightning-thunder version container
3. Install additionally:
```
python3 -m pip install --no-deps huggingface-hub==0.23.2
python3 -m pip install --no-deps transformers==4.40.2
python3 -m pip install -e .
python3 -m pip install git+https://github.com/NVIDIA/Megatron-LM.git@6dd3a1afa4e26d4d27e58d1e83aaa6ee6e36b477
```
4. Execute:
```
rm -f /tmp/graph*.log.txt
export HYDRA_FULL_ERROR=1
export THUNDER_ANNOTATE_TRACES=1
export NEMO_THUNDER_NEVA=dynamo
python3 \
./examples/multimodal/multimodal_llm/neva/neva_pretrain.py \
trainer.precision=bf16-mixed \
model.megatron_amp_O2=True \
model.mcore_gpt=False \
trainer.num_nodes=1 \
trainer.devices=1 \
trainer.val_check_interval=10 \
trainer.limit_val_batches=5 \
trainer.log_every_n_steps=1 \
++exp_manager.max_time_per_run=00:00:03:00 \
trainer.max_steps=20 \
model.micro_batch_size=2 \
model.global_batch_size=4 \
model.tensor_model_parallel_size=1 \
model.pipeline_model_parallel_size=1 \
exp_manager.create_checkpoint_callback=False \
model.data.data_path=./data/multimodal/tiny-neva/dummy.json \
model.data.image_folder=./data/multimodal/tiny-neva/images \
model.tokenizer.library=sentencepiece \
model.tokenizer.model=./data/multimodal/tiny-neva/tokenizer_add_special.model \
model.num_layers=2 \
model.hidden_size=5120 \
model.ffn_hidden_size=13824 \
model.num_attention_heads=40 \
model.normalization=rmsnorm \
model.data.num_workers=0 \
model.data.conv_template=llama_2 \
model.mm_cfg.vision_encoder.from_pretrained=openai/clip-vit-large-patch14 \
model.mm_cfg.llm.from_pretrained=null \
model.use_flash_attention=false \
exp_manager.exp_dir=./nemo_neva
```
### Expected behavior
The pretraining should run smoothly.
### Environment
As in the container
### Additional context
Attaching the full log of the error:
[dynamo_error_2nd_Oct.txt](https://github.com/user-attachments/files/17229666/dynamo_error_2nd_Oct.txt)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.