yi-34b-chat vllm_worker startup failed to register with controller
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
# `vllm_worker` vllm_worker startup failed to register with controller
> vicuna-13b-v1.5-16k can be registered to the controller normally, but yi-34b-chat vllm_worker cannot be registered, and model_worker yi-34b-chat can be registered instead
**Local start**
**Tesla V100S 32G 8 pieces**
## versions
> fastchat
```log
name = "fschat"
version = "0.2.33"
```
> Yi-34b-Chat
## `vllm_worker ` start
```shell
python3 -m fastchat.serve.vllm_worker --model-path=/data01/Yi-34b-Chat --tensor-parallel-size=4 --host=localhost --port=21002 --worker-address=http://localhost:21002 --controller-address=http://localhost:21001 --model-names=Yi-34b-Chat
WARNING 01-04 11:10:33 config.py:286] Casting torch.bfloat16 to torch.float16.
2024-01-04 11:10:35 | ERROR | stderr | 2024-01-04 11:10:35,894 INFO worker.py:1636 -- Started a local Ray instance.
INFO 01-04 11:10:36 llm_engine.py:70] Initializing an LLM engine with config: model='/data01/Yi-34b-Chat', tokenizer='/data01/Yi-34b-Chat', tokenizer_mode=auto, trust_remote_code=True, dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=4, seed=0)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 01-04 11:11:00 llm_engine.py:196] # GPU blocks: 11753, # CPU blocks: 4369
2024-01-04 11:11:03 | INFO | model_worker | Loading the model ['Yi-34b-Chat'] on worker fa0be6ee, worker type: vLLM worker...
2024-01-04 11:11:03 | ERROR | stderr | INFO: Started server process [70946]
2024-01-04 11:11:03 | ERROR | stderr | INFO: Waiting for application startup.
2024-01-04 11:11:03 | ERROR | stderr | INFO: Application startup complete.
2024-01-04 11:11:03 | ERROR | stderr | INFO: Uvicorn running on http://localhost:21002 (Press CTRL+C to quit)
```
## `controller` start
```shell
python3 -m fastchat.serve.controller
2024-01-04 11:15:57 | INFO | controller | args: Namespace(dispatch_method='shortest_queue', host='localhost', port=21001, ssl=False)
2024-01-04 11:15:57 | ERROR | stderr | INFO: Started server process [77651]
2024-01-04 11:15:57 | ERROR | stderr | INFO: Waiting for application startup.
2024-01-04 11:15:57 | ERROR | stderr | INFO: Application startup complete.
2024-01-04 11:15:57 | ERROR | stderr | INFO: Uvicorn running on http://localhost:21001 (Press CTRL+C to quit)
```
How can I solve it,I really need your help
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported vllm_worker and controller entry points and compare the Yi-34b-Chat startup logs with the working vicuna-13b-v1.5-16k case. Trace the registration request from the worker to the controller and identify why the worker does not register; done means Yi-34b-Chat appears in the controller while using vllm_worker.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100