lm-sys / lm-sys/FastChat

yi-34b-chat vllm_worker startup failed to register with controller

Open
#2,882 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

# `vllm_worker` vllm_worker startup failed to register with controller
> vicuna-13b-v1.5-16k can be registered to the controller normally, but yi-34b-chat vllm_worker cannot be registered, and model_worker yi-34b-chat can be registered instead

**Local start**
**Tesla V100S 32G 8 pieces**
## versions

> fastchat
```log
name = "fschat"
version = "0.2.33"
```
> Yi-34b-Chat

## `vllm_worker ` start

```shell
python3 -m fastchat.serve.vllm_worker --model-path=/data01/Yi-34b-Chat --tensor-parallel-size=4 --host=localhost --port=21002 --worker-address=http://localhost:21002 --controller-address=http://localhost:21001 --model-names=Yi-34b-Chat

WARNING 01-04 11:10:33 config.py:286] Casting torch.bfloat16 to torch.float16.
2024-01-04 11:10:35 | ERROR | stderr | 2024-01-04 11:10:35,894 INFO worker.py:1636 -- Started a local Ray instance.
INFO 01-04 11:10:36 llm_engine.py:70] Initializing an LLM engine with config: model='/data01/Yi-34b-Chat', tokenizer='/data01/Yi-34b-Chat', tokenizer_mode=auto, trust_remote_code=True, dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=4, seed=0)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 01-04 11:11:00 llm_engine.py:196] # GPU blocks: 11753, # CPU blocks: 4369
2024-01-04 11:11:03 | INFO | model_worker | Loading the model ['Yi-34b-Chat'] on worker fa0be6ee, worker type: vLLM worker...
2024-01-04 11:11:03 | ERROR | stderr | INFO: Started server process [70946]
2024-01-04 11:11:03 | ERROR | stderr | INFO: Waiting for application startup.
2024-01-04 11:11:03 | ERROR | stderr | INFO: Application startup complete.
2024-01-04 11:11:03 | ERROR | stderr | INFO: Uvicorn running on http://localhost:21002 (Press CTRL+C to quit)
```

## `controller` start
```shell
python3 -m fastchat.serve.controller
2024-01-04 11:15:57 | INFO | controller | args: Namespace(dispatch_method='shortest_queue', host='localhost', port=21001, ssl=False)
2024-01-04 11:15:57 | ERROR | stderr | INFO: Started server process [77651]
2024-01-04 11:15:57 | ERROR | stderr | INFO: Waiting for application startup.
2024-01-04 11:15:57 | ERROR | stderr | INFO: Application startup complete.
2024-01-04 11:15:57 | ERROR | stderr | INFO: Uvicorn running on http://localhost:21001 (Press CTRL+C to quit)
```

How can I solve it,I really need your help

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported vllm_worker and controller entry points and compare the Yi-34b-Chat startup logs with the working vicuna-13b-v1.5-16k case. Trace the registration request from the worker to the controller and identify why the worker does not register; done means Yi-34b-Chat appears in the controller while using vllm_worker.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.