kserve / kserve/open-inference-protocol

Adding a GET Interface to Inference Would Allow for Better Performance

Open
#20 3 comments 2 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
77
Forks
15
PR merge metrics
No merged PRs in 30d

Description

The current specification does not allow for good use of _Cache Control_ i.e., client side caching, which is inefficient in production environments. The specification should add a GET request for inference to allow better use of client side caching with _Cache Control_. Let me explain better.

If a user is querying a deterministic model the response from the endpoint should be the same each time until the model is retrained, at which time the model should get a new version. (For non deterministic models such as simulation the current interface is fine) The current implementation only has a HTTP POST for querying the model for inference. If a HTTP GET is used with proper _Cache Control_ settings the load on the server can be decreased. Cache control allows the client to cache response and the server to control the cache settings. By having the server control the cache other systems such as experimentation can be used on the server side without worry that the client will get the wrong response. The RFC on _Cache Control_ is probably better at explaining this than I am and is included below.

RFC on HTTP caching: [here](https://httpwg.org/specs/rfc9111.html#field.cache-control)

Currently different implementations of this specification use a more inefficient server side caching. Although server side caching can reduce the load on the server, the network bandwidth and round trip delay on the POST request are not eliminated. A good production system should utilize both client side and server side caching to have optimal results.

Here is an example of an implementation that uses server side caching: [here](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/response_cache.html)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the current inference interface in the protocol specification and RFC 9111 on Cache-Control, then compare the existing POST behavior with the proposed GET flow. Done means the specification defines GET semantics and caching behavior for deterministic models without changing the stated behavior for non-deterministic models.

Written by the indexing model from the issue text.

Assessment

Domain
api, backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.