CUDA runtime error when running fastchat.serve.cli to serve Vicuna-7B
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
My GPU is 1x A100 (40 GB SXM4)
I followed the following steps from https://github.com/skypilot-org/skypilot/blob/master/llm/vicuna/serve.yaml before running the command python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.3
conda create -n chatbot python=3.9 -y
conda activate chatbot
# Install dependencies
pip install torch==1.12.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
pip install git+https://github.com/lm-sys/FastChat.git
pip install git+https://github.com/huggingface/transformers.git@41a2f3529c6b56866c317031375ffd3e7b8bea01
The error is
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F,
ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP)`
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the installation steps from llm/vicuna/serve.yaml and reproduce python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.3 on the stated A100 setup. Inspect the reported CUBLAS error and the versions of PyTorch, CUDA, and Transformers involved. Done means identifying a reproducible cause and documenting or implementing a verified resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100