QP request batcher
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 1.2k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 2
Description
/area API
## Describe the feature
### Requirements
* User should be able to specify certain criteria like latency and batch size per service so that QP can decide when to submit the batch to the user container. This would require to annotate a specific service accordingly.
* Each request of the batch is handled independently and transparently. In order to achieve this a protocol should be defined between QP and the user container so that user container should comply with it in order to receive data in batch mode.
* This does not cover the case where requests need to be sent as a batch across the whole Knative data plane
* The feature will be an extension, not enabled by default
### Use cases
* There are scenarios where http requests need to be delivered as a batch instead of one by one. A common scenario is model serving where you get [better performance](https://medium.com/modern-nlp/101-for-serving-ml-models-10217c9f0764) if requests are collected as a batch from the backend in order to apply an operation per data vector and not per data instance. An implementation for Knative Serving that uses an intermediate container can be found in KServe [here](https://kserve.github.io/website/modelserving/batcher/batcher/).
* As discussed [here](https://cloud-native.slack.com/archives/C04LMU0AX60/p1675769223854919) users coming from other systems such as [AWS SQS ](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html) might expect a batch configuration option to consume more than one requests at least at the user container/backend side.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reviewing the requirements for per-service latency and batch-size configuration, the QP-to-user-container protocol, and the opt-in extension constraint. Done would require an agreed design and implementation covering these requirements without batching requests across the whole Knative data plane.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100