Lightning-AI / Lightning-AI/lightning-thunder
HF LLaVa support
@riccardofelluga is already working on this.
Since Sep 19, 2024.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
## 🚀 Model / language coverage
The idea is to support LLaVa model from HF. This issue is mainly for tracking the status.
Blocking issues:
- [ ] #735
- [ ] #124
### Minimal Repro
First of all get the `transformers` library with `pip install transformers` then run this script:
```python
import torch
import thunder
from transformers import LlavaForConditionalGeneration
model = LlavaForConditionalGeneration.from_pretrained(
"llava-hf/llava-1.5-7b-hf",
torch_dtype=torch.bfloat16
)
model.to("cuda")
input_ids = torch.randint(1, 32000, (1, 22), device="cuda")
attention_mask = torch.ones((1, 22), dtype=torch.int64, device="cuda")
pixel_values = torch.randn((1, 3, 336, 336), device="cuda")
labels = torch.randint(-100, 32000, (1, 22), device="cuda")
# Setup fake image id
input_ids[0, 0] = 1
input_ids[0, 5] = 32000
model = thunder.jit(model, executors=thunder.get_default_executors())
out = model(input_ids=input_ids, attention_mask=attention_mask, pixel_values=pixel_values, labels=labels)
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.