Lightning-AI / Lightning-AI/LitServe

Pass headers to litapi

Open
#718 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
3.9k
Forks
304
Avg merge
3d 13h
Merged PRs (30d)
6

Description

## 🚀 Feature

Being able to access headers in the litapi stages (decode, predict, etc.).

### Motivation

- Tracing - user agents, region and similar in distributed logging across stages (decode, predict, etc.)
- Model tweaks (different model, different thresholds, etc.) based on implicit headers (like user agent).

### Pitch

There is benefit to accessing headers natively in litapi that would provide more flexibility in the model inference and execution since headers are being send implicitly by clients and in some cases they can differ the way predict should be executed.

### Alternatives

Currently, we need to repack the request body if we want to add header value to the request object ending up in the litapi.
Or, caller must pass them in body.

### Additional context

https://github.com/Lightning-AI/LitServe/issues/566

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file or test is named. Start by tracing how requests enter litapi and reach the decode and predict stages, then identify where headers are currently discarded. Done means stage code can access the incoming headers without requiring callers to repack them into the request body, with behavior covered by relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.