lm-sys / lm-sys/FastChat

deepseek-coder-33b-instruct model with openai got "InvalidChunkLength" error

Open
#3,283 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

Use FastChat to start the deepseek-coder-33b-instruct model, send a stream request and got an error response.
If set stream=False, you can print a good response
If change to other models, it also works with stream
Start cmd:

python3 -m fastchat.serve.controller
python3 -m fastchat.serve.openai_api_server --host 0.0.0.0 --port 8000
python3 -m fastchat.serve.model_worker --model-path /home/ubuntu/models/deepseek-coder-33b-instruct --num-gpus=2 --gpus=0,1 --max-gpu-memory=46GB --model-names=deepseek-coder-33b-instruct

Below python code for test:

import openai

if __name__ == "__main__":
    openai.api_base = "http://model_addr:8000/v1"
    openai.api_key = "none"

    response = openai.ChatCompletion.create(
        model="deepseek-coder-33b-instruct",
        temperature=0.6,
        top_p=1,
        messages=[{"role": "user", "content": "Hello"}],
        stream=True
    )
    print(response)
    for chunk in response:
        print(chunk, end="", flush=True)

error log:

<generator object EngineAPIResource.create.<locals>.<genexpr> at 0x1030ae900>
Traceback (most recent call last):
  File "/Users/yangglei/miniconda3/envs/my-env/lib/python3.9/site-packages/urllib3/response.py", line 712, in _error_catcher
    yield
  File "/Users/yangglei/miniconda3/envs/my-env/lib/python3.9/site-packages/urllib3/response.py", line 1071, in read_chunked
    self._update_chunk_length()
  File "/Users/yangglei/miniconda3/envs/my-env/lib/python3.9/site-packages/urllib3/response.py", line 1006, in _update_chunk_length
    raise InvalidChunkLength(self, line) from None
urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/yangglei/miniconda3/envs/my-env/lib/python3.9/site-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/Users/yangglei/miniconda3/envs/my-env/lib/python3.9/site-packages/urllib3/response.py", line 931, in stream
    yield from self.read_chunked(amt, decode_content=decode_content)
  File "/Users/yangglei/miniconda3/envs/my-env/lib/python3.9/site-packages/urllib3/response.py", line 1100, in read_chunked
    self._original_response.close()
  File "/Users/yangglei/miniconda3/envs/my-env/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/yangglei/miniconda3/envs/my-env/lib/python3.9/site-packages/urllib3/response.py", line 729, in _error_catcher
    raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/yangglei/work/projects/go/src/python/test-continue-2.py", line 34, in <module>
    for chunk in response:
  File "/Users/yangglei/miniconda3/envs/my-env/lib/python3.9/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 166, in <genexpr>
    return (
  File "/Users/yangglei/miniconda3/envs/my-env/lib/python3.9/site-packages/openai/api_requestor.py", line 692, in <genexpr>
    return (
  File "/Users/yangglei/miniconda3/envs/my-env/lib/python3.9/site-packages/openai/api_requestor.py", line 115, in parse_stream
    for line in rbody:
  File "/Users/yangglei/miniconda3/envs/my-env/lib/python3.9/site-packages/requests/models.py", line 865, in iter_lines
    for chunk in self.iter_content(
  File "/Users/yangglei/miniconda3/envs/my-env/lib/python3.9/site-packages/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the listed controller, OpenAI API server, and model worker commands with deepseek-coder-33b-instruct, then reproduce the streaming request from the Python example. Trace the FastChat OpenAI API streaming path and compare it with a non-streaming request and another model. Done means the affected model streams successfully without the reported chunked-encoding error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.