Error reported when deploying DeepSeek-Coder-V2-Lite-Chat with latest Fastchat

Nobody has claimed this yet.

When using fastchat to deploy the openapi_api interface, calling "/v1/models" returns normal results, but calling "/v1/chat/completions" reports the following errors：

INFO: 192.168.0.2:53398 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
2024-07-02 17:59:32 | INFO | openai_api_server | args: Namespace(host='0.0.0.0', port=8000, controller_address='http://127.0.0.1:21001', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_keys=None, ssl=False)
2024-07-02 17:59:32 | ERROR | stderr | INFO: Started server process [60]
2024-07-02 17:59:32 | ERROR | stderr | INFO: Waiting for application startup.
2024-07-02 17:59:32 | ERROR | stderr | INFO: Application startup complete.
2024-07-02 17:59:32 | ERROR | stderr | INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
2024-07-02 18:00:00 | ERROR | stderr | /usr/local/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings:
2024-07-02 18:00:00 | ERROR | stderr | Expected `str` but got `bool` - serialized value may not be as expected
2024-07-02 18:00:00 | ERROR | stderr | return self.__pydantic_serializer__.to_python(
2024-07-02 18:00:00 | INFO | stdout | INFO: 192.168.0.2:43130 - "GET /v1/models HTTP/1.1" 200 OK
2024-07-02 18:00:00 | INFO | stdout | INFO: 192.168.0.2:43140 - "GET /v1/models HTTP/1.1" 200 OK
2024-07-02 18:00:05 | INFO | stdout | INFO: 192.168.0.2:43148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2024-07-02 18:00:17 | ERROR | stderr | ERROR: Exception in ASGI application
2024-07-02 18:00:17 | ERROR | stderr | Traceback (most recent call last):
2024-07-02 18:00:17 | ERROR | stderr | File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
2024-07-02 18:00:17 | ERROR | stderr | await wrap(partial(self.listen_for_disconnect, receive))
2024-07-02 18:00:17 | ERROR | stderr | File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
2024-07-02 18:00:17 | ERROR | stderr | await func()
2024-07-02 18:00:17 | ERROR | stderr | File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
2024-07-02 18:00:17 | ERROR | stderr | message = await receive()
2024-07-02 18:00:17 | ERROR | stderr | File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
2024-07-02 18:00:17 | ERROR | stderr | await self.message_event.wait()
2024-07-02 18:00:17 | ERROR | stderr | File "/usr/local/lib/python3.10/asyncio/locks.py", line 214, in wait
2024-07-02 18:00:17 | ERROR | stderr | await fut
2024-07-02 18:00:17 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 14cf83173760
2024-07-02 18:00:17 | ERROR | stderr |
2024-07-02 18:00:17 | ERROR | stderr | During handling of the above exception, another exception occurred:
2024-07-02 18:00:17 | ERROR | stderr |
2024-07-02 18:00:17 | ERROR | stderr | + Exception Group Traceback (most recent call last):
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
2024-07-02 18:00:17 | ERROR | stderr | | result = await app( # type: ignore[func-returns-value]
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
2024-07-02 18:00:17 | ERROR | stderr | | return await self.app(scope, receive, send)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
2024-07-02 18:00:17 | ERROR | stderr | | await super().__call__(scope, receive, send)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
2024-07-02 18:00:17 | ERROR | stderr | | await self.middleware_stack(scope, receive, send)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
2024-07-02 18:00:17 | ERROR | stderr | | raise exc
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
2024-07-02 18:00:17 | ERROR | stderr | | await self.app(scope, receive, _send)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__
2024-07-02 18:00:17 | ERROR | stderr | | await self.app(scope, receive, send)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
2024-07-02 18:00:17 | ERROR | stderr | | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
2024-07-02 18:00:17 | ERROR | stderr | | raise exc
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2024-07-02 18:00:17 | ERROR | stderr | | await app(scope, receive, sender)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
2024-07-02 18:00:17 | ERROR | stderr | | await self.middleware_stack(scope, receive, send)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
2024-07-02 18:00:17 | ERROR | stderr | | await route.handle(scope, receive, send)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
2024-07-02 18:00:17 | ERROR | stderr | | await self.app(scope, receive, send)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
2024-07-02 18:00:17 | ERROR | stderr | | await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
2024-07-02 18:00:17 | ERROR | stderr | | raise exc
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2024-07-02 18:00:17 | ERROR | stderr | | await app(scope, receive, sender)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
2024-07-02 18:00:17 | ERROR | stderr | | await response(scope, receive, send)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
2024-07-02 18:00:17 | ERROR | stderr | | async with anyio.create_task_group() as task_group:
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
2024-07-02 18:00:17 | ERROR | stderr | | raise BaseExceptionGroup(
2024-07-02 18:00:17 | ERROR | stderr | | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
2024-07-02 18:00:17 | ERROR | stderr | +-+---------------- 1 ----------------
2024-07-02 18:00:17 | ERROR | stderr | | Traceback (most recent call last):
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
2024-07-02 18:00:17 | ERROR | stderr | | yield
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 254, in __aiter__
2024-07-02 18:00:17 | ERROR | stderr | | async for part in self._httpcore_stream:
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 367, in __aiter__
2024-07-02 18:00:17 | ERROR | stderr | | raise exc from None
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 363, in __aiter__
2024-07-02 18:00:17 | ERROR | stderr | | async for part in self._stream:
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 349, in __aiter__
2024-07-02 18:00:17 | ERROR | stderr | | raise exc
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 341, in __aiter__
2024-07-02 18:00:17 | ERROR | stderr | | async for chunk in self._connection._receive_response_body(**kwargs):
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 210, in _receive_response_body
2024-07-02 18:00:17 | ERROR | stderr | | event = await self._receive_event(timeout=timeout)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 220, in _receive_event
2024-07-02 18:00:17 | ERROR | stderr | | with map_exceptions({h11.RemoteProtocolError: RemoteProtocolError}):
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
2024-07-02 18:00:17 | ERROR | stderr | | self.gen.throw(typ, value, traceback)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
2024-07-02 18:00:17 | ERROR | stderr | | raise to_exc(exc) from exc
2024-07-02 18:00:17 | ERROR | stderr | | httpcore.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)
2024-07-02 18:00:17 | ERROR | stderr | |
2024-07-02 18:00:17 | ERROR | stderr | | The above exception was the direct cause of the following exception:
2024-07-02 18:00:17 | ERROR | stderr | |
2024-07-02 18:00:17 | ERROR | stderr | | Traceback (most recent call last):
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
2024-07-02 18:00:17 | ERROR | stderr | | await func()
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 250, in stream_response
2024-07-02 18:00:17 | ERROR | stderr | | async for chunk in self.body_iterator:
2024-07-02 18:00:17 | ERROR | stderr | | File "/app/FastChat-main/fastchat/serve/openai_api_server.py", line 508, in chat_completion_stream_generator
2024-07-02 18:00:17 | ERROR | stderr | | async for content in generate_completion_stream(gen_params, worker_addr):
2024-07-02 18:00:17 | ERROR | stderr | | File "/app/FastChat-main/fastchat/serve/openai_api_server.py", line 693, in generate_completion_stream
2024-07-02 18:00:17 | ERROR | stderr | | async for raw_chunk in response.aiter_raw():
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/httpx/_models.py", line 987, in aiter_raw
2024-07-02 18:00:17 | ERROR | stderr | | async for raw_stream_bytes in self.stream:
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 149, in __aiter__
2024-07-02 18:00:17 | ERROR | stderr | | async for chunk in self._stream:
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 253, in __aiter__
2024-07-02 18:00:17 | ERROR | stderr | | with map_httpcore_exceptions():
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
2024-07-02 18:00:17 | ERROR | stderr | | self.gen.throw(typ, value, traceback)
2024-07-02 18:00:17 | ERROR | stderr | | File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
2024-07-02 18:00:17 | ERROR | stderr | | raise mapped_exc(message) from exc
2024-07-02 18:00:17 | ERROR | stderr | | httpx.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)
2024-07-02 18:00:17 | ERROR | stderr | +------------------------------------

Does fastchat not support DeepSeek-V2 model at the moment?

Reproduce the DeepSeek-Coder-V2-Lite-Chat deployment and call /v1/chat/completions, then inspect fastchat/serve/openai_api_server.py at chat_completion_stream_generator (line 508) and generate_completion_stream (line 693). Trace the streamed response handling around response.aiter_raw(); done means the request no longer ends with the incomplete chunked read and returns a complete chat completion.

Written by the indexing model from the issue text.