sglang_worker doesn't work for GPTQ models
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
I tried to load a GPTQ model with sglang_worker, but failed. Error message as below:
```
2024-03-03 15:15:25 | INFO | stdout | File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/router/manager.py", line 68, in start_router_process
2024-03-03 15:15:25 | INFO | stdout | model_client = ModelRpcClient(server_args, port_args)
2024-03-03 15:15:25 | INFO | stdout | File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/router/model_rpc.py", line 564, in __init__
2024-03-03 15:15:25 | INFO | stdout | self.model_servers = [x[0] for x in rets]
2024-03-03 15:15:25 | INFO | stdout | File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/router/model_rpc.py", line 564, in
2024-03-03 15:15:25 | INFO | stdout | self.model_servers = [x[0] for x in rets]
2024-03-03 15:15:25 | INFO | stdout | File "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
2024-03-03 15:15:25 | INFO | stdout | yield _result_or_cancel(fs.pop())
2024-03-03 15:15:25 | INFO | stdout | File "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
2024-03-03 15:15:25 | INFO | stdout | return fut.result(timeout)
2024-03-03 15:15:25 | INFO | stdout | File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2024-03-03 15:15:25 | INFO | stdout | return self.__get_result()
2024-03-03 15:15:25 | INFO | stdout | File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2024-03-03 15:15:25 | INFO | stdout | raise self._exception
2024-03-03 15:15:25 | INFO | stdout | File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
2024-03-03 15:15:25 | INFO | stdout | result = self.fn(*self.args, **self.kwargs)
2024-03-03 15:15:25 | INFO | stdout | File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/router/model_rpc.py", line 597, in start_model_process
2024-03-03 15:15:25 | INFO | stdout | proc.start()
2024-03-03 15:15:25 | INFO | stdout | File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
2024-03-03 15:15:25 | INFO | stdout | self._popen = self._Popen(self)
2024-03-03 15:15:25 | INFO | stdout | File "/usr/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
2024-03-03 15:15:25 | INFO | stdout | return _default_context.get_context().Process._Popen(process_obj)
2024-03-03 15:15:25 | INFO | stdout | File "/usr/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
2024-03-03 15:15:25 | INFO | stdout | return Popen(process_obj)
2024-03-03 15:15:25 | INFO | stdout | File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
2024-03-03 15:15:25 | INFO | stdout | super().__init__(process_obj)
2024-03-03 15:15:25 | INFO | stdout | File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
2024-03-03 15:15:25 | INFO | stdout | self._launch(process_obj)
2024-03-03 15:15:25 | INFO | stdout | File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
2024-03-03 15:15:25 | INFO | stdout | reduction.dump(process_obj, fp)
2024-03-03 15:15:25 | INFO | stdout | File "/usr/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
2024-03-03 15:15:25 | INFO | stdout | ForkingPickler(file, protocol).dump(obj)
2024-03-03 15:15:25 | INFO | stdout | AttributeError: Can't pickle local object 'start_model_process.._init_service'
```
My command to start sglang_worker:
```shell
python3 -m fastchat.serve.sglang_worker \
--model-path /opt/models/fastchat/openbuddy-mixtral-7bx8-v18.1-32k-gptq \
--tokenizer-path /opt/models/fastchat/openbuddy-mixtral-7bx8-v18.1-32k-gptq \
--controller-address http://$(CONTROLLER-HOST):$(CONTROLLER-PORT) \
--worker-address http://fastchat-model:21002 \
--num-gpus 2
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the shown python -m fastchat.serve.sglang_worker command and the GPTQ model paths. Start by tracing fastchat.serve.sglang_worker and the referenced sglang manager files, especially model_rpc.py around start_model_process; done means the worker starts successfully with the supplied GPTQ model and no longer raises the pickling error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100