[Feature] 海光DCU简单测试,希望能支持
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
### Motivation
我希望在海光的DCU上能拉起服务
```bash
lmdeploy serve api_server /data/models/qwen/Qwen2-7B-Instruct \
> --model-name Qwen2-7B-Instruct \
> --server-port 8000 \
> --tp 2
2024-08-27 16:11:39,097 - lmdeploy - WARNING - Fallback to pytorch engine because turbomind engine is not installed correctly. If you insist to use turbomind engine, you may need to reinstall lmdeploy from pypi or build from source and try again.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0827 16:11:47.257916 14849 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 3075840000000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=93939288346384
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0827 16:11:47.380654 14979 ProcessGroupNCCL.cpp:686] [Rank 1] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 3075840000000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=94797938383200
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 6.27it/s]
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:01<00:00, 3.89it/s]
I0827 16:11:55.645321 14849 ProcessGroupNCCL.cpp:1340] NCCL_DEBUG: N/A
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
HINT: Please open http://0.0.0.0:8000 in a browser for detailed api usage!!!
HINT: Please open http://0.0.0.0:8000 in a browser for detailed api usage!!!
HINT: Please open http://0.0.0.0:8000 in a browser for detailed api usage!!!
INFO: Started server process [14849]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
```
但当我访问服务时:
```bash
curl "http://localhost:8000/v1/chat/completions" \
> -H "Content-Type: application/json" \
> -H "Authorization: Bearer replace_api_key" \
> -d '{
> "messages": [
> {
> "role": "system",
> "content": "You are a helpful assistant."
> },
> {
> "role": "user",
> "content": "你好,你是谁?你会做什么?"
> }
> ]
> }'
```
服务端coredump了
```
Unsupported conversion from bf16 to f16
UNREACHABLE executed at ../../../lib/Conversion/TritonGPUToLLVM/ElementwiseOpToLLVM.cpp:1798!
Unsupported conversion from bf16 to f16
UNREACHABLE executed at ../../../lib/Conversion/TritonGPUToLLVM/ElementwiseOpToLLVM.cpp:1798!
Aborted (core dumped)
root@sealos:/lmdeploy#
root@sealos:/lmdeploy# /usr/local/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
```
### Related resources
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.