knative / knative/serving

QP request batcher

Open
#13,691 2 comments 0 reactions 0 assignees View on GitHub
area/API kind/feature triage/accepted
Dominant language
Go
Stars
6.1k
Forks
1.2k
Avg merge
2d 7h
Merged PRs (30d)
2

Description

/area API

## Describe the feature

### Requirements

* User should be able to specify certain criteria like latency and batch size per service so that QP can decide when to submit the batch to the user container. This would require to annotate a specific service accordingly.
* Each request of the batch is handled independently and transparently. In order to achieve this a protocol should be defined between QP and the user container so that user container should comply with it in order to receive data in batch mode.
* This does not cover the case where requests need to be sent as a batch across the whole Knative data plane
* The feature will be an extension, not enabled by default

### Use cases

* There are scenarios where http requests need to be delivered as a batch instead of one by one. A common scenario is model serving where you get [better performance](https://medium.com/modern-nlp/101-for-serving-ml-models-10217c9f0764) if requests are collected as a batch from the backend in order to apply an operation per data vector and not per data instance. An implementation for Knative Serving that uses an intermediate container can be found in KServe [here](https://kserve.github.io/website/modelserving/batcher/batcher/).
* As discussed [here](https://cloud-native.slack.com/archives/C04LMU0AX60/p1675769223854919) users coming from other systems such as [AWS SQS ](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html) might expect a batch configuration option to consume more than one requests at least at the user container/backend side.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing the requirements for per-service latency and batch-size configuration, the QP-to-user-container protocol, and the opt-in extension constraint. Done would require an agreed design and implementation covering these requirements without batching requests across the whole Knative data plane.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.