microsoft / microsoft/onnxruntime-genai

Benchmark script crashes when trying to run Llama 3.2 1b DML model

Open
#982 0 comments 0 reactions 1 assignee Claimed by @PatriceVignola View on GitHub
crash ep:DML
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

1. Download or generate the Llama 3.2 1b DML model

2. Install `onnxruntime-genai-directml` version `0.4.0`

3. Run the following command

```
python benchmark_e2e.py -i C:\Users\nakersha\Develop\models\meta-llama\Llama-3.2-1b\directml-int4-awq-block-128 -v
```

4. Output is as follows ie crashes with no error message

```
Args: batch_size = 1, prompt_length = 16, tokens = 256, max_length = 272
Loading model...
Model loaded
Running warmup runs...
0%| | 0/5 [00:00
```
- OS: Windows
- Python 3.10.15
- Conda env

```
Package Version
-------------------------- -----------
alembic 1.13.3
annotated-types 0.7.0
certifi 2024.8.30
charset-normalizer 3.4.0
colorama 0.4.6
coloredlogs 15.0.1
colorlog 6.8.2
filelock 3.16.1
flatbuffers 24.3.25
fsspec 2024.9.0
greenlet 3.1.1
huggingface-hub 0.25.2
humanfriendly 10.0
idna 3.10
Jinja2 3.1.4
lightning-utilities 0.11.7
Mako 1.3.5
MarkupSafe 3.0.1
mpmath 1.3.0
networkx 3.4.1
numpy 2.1.2
olive-ai 0.6.2
onnx 1.17.0
onnxruntime-directml 1.19.2
onnxruntime-genai-directml 0.4.0
optuna 4.0.0
packaging 24.1
pandas 2.2.3
pip 24.2
protobuf 3.20.3
psutil 6.0.0
pydantic 2.9.2
pydantic_core 2.23.4
pyreadline3 3.5.4
python-dateutil 2.9.0.post0
pytz 2024.2
PyYAML 6.0.2
regex 2024.9.11
requests 2.32.3
safetensors 0.4.5
setuptools 75.1.0
six 1.16.0
SQLAlchemy 2.0.35
sympy 1.13.3
tokenizers 0.20.1
torch 2.4.1
torchmetrics 1.4.3
tqdm 4.66.5
transformers 4.45.2
typing_extensions 4.12.2
tzdata 2024.2
urllib3 2.2.3
wheel 0.44.0
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.