Lightning-AI / Lightning-AI/LitServe

Pass binary payloads to `decode_request` unchanged

Open
#732 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
3.9k
Forks
304
Avg merge
3d 13h
Merged PRs (30d)
6

Description

## 🚀 Feature

Pass binary payloads to `decode_request` unchanged.

### Motivation

We use LitServe for BYOC SageMaker Inference endpoints that take protobuf requests of the following format:

```http
POST /invocations
Content-Type: application/x-recordio-protobuf

```

LitServe currently parses all non-form/multipart request requests as JSON. To support `application/x-recordio-protobuf`, we override private methods:

```python

class _RawRequestHandler(RegularRequestHandler):
"""Read the raw protobuf body instead of parsing JSON. The returned dict (bytes + str)
is picklable, so it crosses the worker transport cleanly."""

async def _prepare_request(
self, request: Request, request_type: Any
) -> dict[str, Any]:
return {
"body": await request.body(),
"content_type": request.headers.get("Content-Type", ""),
}

class RankerLitServer(ls.LitServer):
def _register_api_endpoints(
self, lit_api: ls.LitAPI, request_type: Any, response_type: Any
) -> None:
handler = _RawRequestHandler(lit_api, self)

async def endpoint_handler(request: Request) -> Response:
return await handler.handle_request(request, Request)

self.app.add_api_route(
lit_api.api_path,
endpoint_handler,
methods=["POST"],
dependencies=[Depends(self.setup_auth())],
)
```

While it works, overriding private APIs makes future LitServe upgrades brittle.

### Pitch

Enable LitServe to pass bytes from the request body to `decode_request`. This can either be opt-in manually, or via introspecting the `Content-type` header. For raw byte bodies, LitServe
should:

1. Read the request via `await request.body()` without JSON parsing.
2. Pass the resulting `bytes` to `decode_request`.
3. Make the request content type available to the API.
4. Preserve the existing worker-transport and batching behavior.
5. Leave existing JSON and multipart handling unchanged by default.

### Alternatives

- **Private handler override:** This is our current solution, but it relies on
internal methods that may change between LitServe releases.

### Additional context

Past issue #362

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the request path through _register_api_endpoints, _prepare_request, handle_request, and decode_request, including the existing JSON and multipart handling. Check how worker transport and batching are preserved, then verify that binary request bodies and their Content-Type reach the API unchanged while existing defaults remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
fastapi, python
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.