lm-sys / lm-sys/FastChat

NETWORK ERROR DUE TO HIGH TRAFFIC. PLEASE REGENERATE OR REFRESH THIS PAGE

Open
#3,042 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
No merged PRs in 30d

Description

Load model through worker and using fastchat ask OpenAI API.

When the model was first loaded, Memory was about 8G/24G. As API was accessed, the memory usage kept growing. In the end, NETWORK ERROR DUE TO HIGH TRAFFIC. PLEASE REGENERATE OR REFRESH THIS PAGE showed up.

BadRequestError: Error code: 400 - {'object': 'error', 'message': '**NETWORK ERROR DUE TO HIGH TRAFFIC. PLEASE REGENERATE OR REFRESH THIS PAGE.**\n\n(CUDA out of memory. Tried to allocate 26.00 MiB. GPU 0 has a total capacty of 23.69 GiB of which 1.94 MiB is free. Including non-PyTorch memory, this process has 23.66 GiB memory in use. Of the allocated memory 22.03 GiB is allocated by PyTorch, and 1.31 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF)', 'code': 50002}

image

(fastchat-3.9) user@R750B~/HuggingFace/Qwen/Qwen1.5-14B-Chat-AWQ $ python3 -m fastchat.serve.model_worker --model-name chat --model-path ./ --num-gpus 2 --load-8bit --awq-wbits 4
2024-02-13 21:42:23 | INFO | model_worker | args: Namespace(host='localhost', port=21002, worker_address='http://localhost:21002', controller_address='http://localhost:21001', model_path='./', revision='main', device='cuda', gpus=None, num_gpus=2, max_gpu_memory=None, dtype=None, load_8bit=True, cpu_offloading=False, gptq_ckpt=None, gptq_wbits=16, gptq_groupsize=-1, gptq_act_order=False, awq_ckpt=None, awq_wbits=4, awq_groupsize=-1, enable_exllama=False, exllama_max_seq_len=4096, exllama_gpu_split=None, exllama_cache_8bit=False, enable_xft=False, xft_max_seq_len=4096, xft_dtype=None, model_names=['chat'], conv_template=None, embed_in_truncate=False, limit_worker_concurrency=5, stream_interval=2, no_register=False, seed=None, debug=False, ssl=False)
2024-02-13 21:42:23 | INFO | model_worker | Loading the model ['chat'] on worker 04a31e1d ...
2024-02-13 21:42:23 | ERROR | stderr | /home/user/tmp/FastChat/fastchat/model/model_adapter.py:277: UserWarning: 8-bit quantization is not supported for multi-gpu inference.
2024-02-13 21:42:23 | ERROR | stderr |   warnings.warn(
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-02-13 21:42:24 | INFO | numexpr.utils | Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-02-13 21:42:24 | INFO | numexpr.utils | NumExpr defaulting to 8 threads.
2024-02-13 21:42:24 | INFO | datasets | PyTorch version 2.1.2+cu121 available.
Loading checkpoint shards:   0%|                                                                                                                                                        | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:  33%|████████████████████████████████████████████████                                                                                                | 1/3 [00:35<01:10, 35.05s/it]
Loading checkpoint shards:  67%|████████████████████████████████████████████████████████████████████████████████████████████████                                                | 2/3 [01:10<00:35, 35.29s/it]
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [01:25<00:00, 25.92s/it]
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [01:25<00:00, 28.43s/it]
2024-02-13 21:43:50 | ERROR | stderr |
2024-02-13 21:43:50 | INFO | model_worker | Register to controller
2024-02-13 21:43:50 | ERROR | stderr | INFO:     Started server process [1993665]
2024-02-13 21:43:50 | ERROR | stderr | INFO:     Waiting for application startup.
2024-02-13 21:43:50 | ERROR | stderr | INFO:     Application startup complete.
2024-02-13 21:43:50 | ERROR | stderr | INFO:     Uvicorn running on http://localhost:21002 (Press CTRL+C to quit)
2024-02-13 21:44:35 | INFO | model_worker | Send heart beat. Models: ['chat']. Semaphore: None. call_ct: 0. worker_id: 04a31e1d.
2024-02-13 21:45:20 | INFO | model_worker | Send heart beat. Models: ['chat']. Semaphore: None. call_ct: 0. worker_id: 04a31e1d.
......
2024-02-13 21:58:19 | INFO | stdout | INFO:     127.0.0.1:58710 - "POST /count_token HTTP/1.1" 200 OK
2024-02-13 21:58:20 | INFO | stdout | INFO:     127.0.0.1:58718 - "POST /worker_generate HTTP/1.1" 200 OK
2024-02-13 21:58:20 | INFO | stdout | INFO:     127.0.0.1:36042 - "POST /model_details HTTP/1.1" 200 OK
2024-02-13 21:58:20 | INFO | stdout | INFO:     127.0.0.1:36050 - "POST /count_token HTTP/1.1" 200 OK
2024-02-13 21:58:21 | INFO | stdout | INFO:     127.0.0.1:36052 - "POST /worker_generate HTTP/1.1" 200 OK
2024-02-13 21:58:23 | INFO | stdout | INFO:     127.0.0.1:36068 - "POST /model_details HTTP/1.1" 200 OK
2024-02-13 21:58:23 | INFO | stdout | INFO:     127.0.0.1:36070 - "POST /count_token HTTP/1.1" 200 OK
2024-02-13 21:58:51 | INFO | model_worker | Send heart beat. Models: ['chat']. Semaphore: Semaphore(value=3, locked=False). call_ct: 59. worker_id: 04a31e1d.
2024-02-13 21:59:36 | INFO | model_worker | Send heart beat. Models: ['chat']. Semaphore: Semaphore(value=3, locked=False). call_ct: 59. worker_id: 04a31e1d.
2024-02-13 22:00:21 | INFO | model_worker | Send heart beat. Models: ['chat']. Semaphore: Semaphore(value=3, locked=False). call_ct: 59. worker_id: 04a31e1d.
2024-02-13 22:00:57 | INFO | stdout | INFO:     127.0.0.1:49526 - "POST /worker_generate HTTP/1.1" 200 OK
2024-02-13 22:00:57 | INFO | stdout | INFO:     127.0.0.1:36082 - "POST /worker_generate HTTP/1.1" 200 OK
2024-02-13 22:01:06 | INFO | model_worker | Send heart beat. Models: ['chat']. Semaphore: Semaphore(value=5, locked=False). call_ct: 59. worker_id: 04a31e1d.
2024-02-13 22:01:52 | INFO | model_worker | Send heart beat. Models: ['chat']. Semaphore: Semaphore(value=5, locked=False). call_ct: 59. worker_id: 04a31e1d.
2024-02-13 22:02:37 | INFO | model_worker | Send heart beat. Models: ['chat']. Semaphore: Semaphore(value=5, locked=False). call_ct: 59. worker_id: 04a31e1d.
2024-02-13 22:03:22 | INFO | model_worker | Send heart beat. Models: ['chat']. Semaphore: Semaphore(value=5, locked=False). call_ct: 59. worker_id: 04a31e1d.
2024-02-13 22:04:07 | INFO | model_worker | Send heart beat. Models: ['chat']. Semaphore: Semaphore(value=5, locked=False). call_ct: 59. worker_id: 04a31e1d.
2024-02-13 22:04:52 | INFO | model_worker | Send heart beat. Models: ['chat']. Semaphore: Semaphore(value=5, locked=False). call_ct: 59. worker_id: 04a31e1d.

torch 2.1.2
cuda 12.1
fschat 0.2.36
awq 0.1.0

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the fastchat.serve.model_worker entry point and the repeated /worker_generate requests shown in the logs. Reproduce the issue with the reported multi-GPU command and monitor memory across requests; done means repeated generation no longer grows memory until CUDA out-of-memory errors occur.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.