Kuadrant / Kuadrant/authorino-operator

Make authorization service headless to enable gRPC load distribution across replicas

Open
#362 0 comments 0 reactions 0 assignees View on GitHub
area/implementation kind/bug
Dominant language
Go
Stars
17
Forks
32
Avg merge
2d 20h
Merged PRs (30d)
10

Description

## Problem

When Authorino is deployed with multiple replicas, the `-authorino-authorization` Kubernetes Service is a standard ClusterIP service. The Kuadrant operator's EnvoyFilter configures Envoy with a `STRICT_DNS` cluster pointing to this service's FQDN. Because a ClusterIP service resolves to a single virtual IP, Envoy creates one upstream endpoint and maintains one persistent HTTP/2 connection to it. All multiplexed gRPC ext_authz streams ride this single connection, which kube-proxy forwards to the same Authorino pod regardless of replica count.

**Observed behavior with 3 Authorino replicas under load:**
- One pod received zero requests across a full load test
- The other two received ~250 and ~238 requests respectively
- Above ~25 concurrent requests, `UNAVAILABLE` (gRPC code 14) errors appear despite healthy replicas being available

## Root Cause

```
STRICT_DNS cluster → ClusterIP FQDN → single VIP → single upstream host
→ single persistent HTTP/2 connection → all gRPC streams pinned to 1–2 pods
→ ROUND_ROBIN lb_policy has nothing to round-robin against
```

## Fix

Make the `authorino-authorization` service headless (`clusterIP: None`). DNS then returns one A record per pod IP, giving Envoy N upstream hosts for N replicas. `ROUND_ROBIN` distributes each new gRPC stream across N connections (one per pod), achieving true stream-level load distribution.

The reconciler must handle the ClusterIP immutability constraint: an existing ClusterIP service cannot be patched to headless — it must be deleted and recreated.

## Companion fix

kuadrant/kuadrant-operator — adds `dns_refresh_rate` and `outlier_detection` to the ext_authz cluster so pod IP churn during rolling updates is handled gracefully.

Contributor guide

Open the contributing guide

Research direction

Start with the reconciler that creates the authorino-authorization Kubernetes Service and trace how it handles existing Services. Verify the headless Service gives DNS records for each Authorino replica and that an existing ClusterIP Service is deleted and recreated when necessary. Done means the generated Service is headless and reconciliation remains safe during updates.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc, kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.