microsoft / microsoft/onnxruntime-genai
Benchmark script crashes when trying to run Llama 3.2 1b DML model
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
1. Download or generate the Llama 3.2 1b DML model
2. Install `onnxruntime-genai-directml` version `0.4.0`
3. Run the following command
```
python benchmark_e2e.py -i C:\Users\nakersha\Develop\models\meta-llama\Llama-3.2-1b\directml-int4-awq-block-128 -v
```
4. Output is as follows ie crashes with no error message
```
Args: batch_size = 1, prompt_length = 16, tokens = 256, max_length = 272
Loading model...
Model loaded
Running warmup runs...
0%| | 0/5 [00:00
```
- OS: Windows
- Python 3.10.15
- Conda env
```
Package Version
-------------------------- -----------
alembic 1.13.3
annotated-types 0.7.0
certifi 2024.8.30
charset-normalizer 3.4.0
colorama 0.4.6
coloredlogs 15.0.1
colorlog 6.8.2
filelock 3.16.1
flatbuffers 24.3.25
fsspec 2024.9.0
greenlet 3.1.1
huggingface-hub 0.25.2
humanfriendly 10.0
idna 3.10
Jinja2 3.1.4
lightning-utilities 0.11.7
Mako 1.3.5
MarkupSafe 3.0.1
mpmath 1.3.0
networkx 3.4.1
numpy 2.1.2
olive-ai 0.6.2
onnx 1.17.0
onnxruntime-directml 1.19.2
onnxruntime-genai-directml 0.4.0
optuna 4.0.0
packaging 24.1
pandas 2.2.3
pip 24.2
protobuf 3.20.3
psutil 6.0.0
pydantic 2.9.2
pydantic_core 2.23.4
pyreadline3 3.5.4
python-dateutil 2.9.0.post0
pytz 2024.2
PyYAML 6.0.2
regex 2024.9.11
requests 2.32.3
safetensors 0.4.5
setuptools 75.1.0
six 1.16.0
SQLAlchemy 2.0.35
sympy 1.13.3
tokenizers 0.20.1
torch 2.4.1
torchmetrics 1.4.3
tqdm 4.66.5
transformers 4.45.2
typing_extensions 4.12.2
tzdata 2024.2
urllib3 2.2.3
wheel 0.44.0
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.