🐛 [Bug] Half precision index tensor detected when converting `mistralai/Mistral-7B-v0.3`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3k
- Forks
- 410
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 78
Description
Bug Description
mistralai/Mistral-7B-v0.3 is not fully supported because the index inputs to torch.ops.aten.index.Tensor are half-precision according to the fake tensor metadata.
To Reproduce
torch.compile and run inference on the mistralai/Mistral-7B-v0.3 model
Expected behavior
The graph consists of 2895 Total Operators, of which 2831 operators are supported, 97.79% coverage
The following ops are currently unsupported or excluded from conversion, and are listed with their op-count in the graph:
torch.ops.aten.index.Tensor: 64
Environment
Build information about Torch-TensorRT can be found by turning on debug messages
- Torch-TensorRT Version (e.g. 1.0.0):
2.3.0 - PyTorch Version (e.g. 1.0):
2.3.1+cu121 - Transformers Version:
4.40.2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with torch.compile on mistralai/Mistral-7B-v0.3 using the reported PyTorch, Transformers, and Torch-TensorRT versions. Inspect the fake tensor metadata for torch.ops.aten.index.Tensor and compare the resulting operator coverage with the reported 2,895-operator graph; done means the half-precision index inputs no longer prevent the model conversion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- compilers, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100