kubernetes-sigs / kubernetes-sigs/gateway-api

Add API for probing metrics from Pods

Open
#4,701 4 comments 0 reactions 1 assignee Claimed by @DamianSawicki View on GitHub
kind/feature lifecycle/rotten
Dominant language
Go
Stars
3k
Forks
789
Avg merge
2d 15h
Merged PRs (30d)
45

Description

**What would you like to be added**:

I'd like to introduce API to configure the gateway probing pods for their metrics. Probing can be asynchronous or synchronous. Asynchronous probing is most appropriate for high qps systems and synchronous probing for long running requests (like inference), where probe latency is insignificant. Equipped with the latest metric values, the gateway can make superior load balancing decisions.

Many Gateway API implementations actively probe backend Pods for their _health_. Some implementations support probing for _load-balancing_ purposes:
* [Agent Checks](https://www.haproxy.com/documentation/haproxy-configuration-tutorials/reliability/health-checks/#agent-checks) in HAProxy, where `maxconn` returned in response to the probe can be [dynamically computed](https://www.haproxy.com/documentation/haproxy-configuration-tutorials/reliability/health-checks/#create-an-agent-program) based on parameters like CPU load,
* [Client-Side Weighted Round Robin Load Balancing Policy](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/load_balancing_policies/client_side_weighted_round_robin/v3/client_side_weighted_round_robin.proto) in Envoy, which allows probing servers (on top of in-band reporting in responses) for a user-specified utilization metric .

The present issue aims at a general API for metric probing, independent from specific load balancing algorithms relying on such probe results. It should support standard metric protocols like Prometheus and ORCA. This parallels a similar effort in the Envoy community https://github.com/envoyproxy/envoy/issues/43915.

**Why this is needed**:

As proved in the [paper](https://www.usenix.org/conference/nsdi24/presentation/wydrowski) "Load is not what you should balance: Introducing Prequal", load balancing based on current probes of the latency and the number of requests in flight can dramatically decrease tail latency, error rates, and resource use. Such an approach is particularly useful for heterogeneous (possibly time-varying) server capacities and requests with highly-variable processing times like inference.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.