huggingface / huggingface/optimum-nvidia

Can't Run README Code

Open
#127 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1k
Forks
103
Avg merge
2m
Merged PRs (30d)
1

Description

Hello! I followed the instructions provided in the README file:

I created a new environment and ran this:

```
apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev
python -m pip install --pre --extra-index-url https://pypi.nvidia.com optimum-nvidia
```

Then ran the sample code on Llama2 provided in the README and got the following error:

```
RuntimeError: [TensorRT-LLM][ERROR] Assertion failed: Failed to deserialize cuda engine (/home/jenkins/agent/workspace/LLM/main/L0_MergeRequest/tensorrt_llm/cpp/tensorrt_llm/runtime/tllmRuntime.cpp:68)
```

Any clue why is that ? By the way nvcc --version is 12.1.

Note:

- The pip install ended up installing a CPU version of torch so I reinstalled it using
`pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 --index-url https://download.pytorch.org/whl/cu121`

- I'm running on A100 GPUs

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.