envoyproxy / envoyproxy/gateway

Add `EnvoyProxy.spec.runtime` to configure Envoy runtime values, served over RTDS

Open
#9,670 1 comment 0 reactions 0 assignees View on GitHub
area/api area/xds-translator
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

*Description*:

Envoy Gateway has no API for setting Envoy runtime values. The only way to reach `layered_runtime` is `spec.bootstrap`, which means raw bootstrap patching and a proxy restart for every change. This proposes a typed `EnvoyProxy.spec.runtime` field, served to proxies over RTDS so values can be changed without a restart.

**Proposal**

1. Add `EnvoyProxy.spec.runtime`, a map of runtime keys to values.
2. Restore an `rtds_layer` to the default bootstrap, layered above `global_config`.
3. Have the xDS translator emit an `envoy.service.runtime.v3.Runtime` resource built from `spec.runtime`, always present in the snapshot even when empty.

Editing `spec.runtime` then pushes a new Runtime resource over the existing ADS stream. No restart, no CDS churn.

Most of the server side already exists. `internal/xds/runner/runner.go` registers `runtimev3.RegisterRuntimeDiscoveryServiceServer`, and go-control-plane's snapshot cache already handles `resource.RuntimeType`.

**Alternatives considered**

- *`spec.runtime` backed by a static layer instead of RTDS.* Same API, but bootstrap is read
at startup, so every change needs a proxy restart.

**Scope**

This adds a way to override runtime-gated settings. It does not:

- make extension-created clusters visible to `BackendTrafficPolicy`
- reach cluster settings with no runtime key, such as `connect_timeout` or TLS config
- make arbitrary cluster edits non-disruptive; the no-reset property is specific to values
Envoy reads from runtime

**Workaround available today**

`spec.bootstrap` with `type: JSONPatch` can set the key into the existing `global_config`
static layer, with no changes to Envoy Gateway:

```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: envoy-gateway-system
spec:
bootstrap:
type: JSONPatch
jsonPatches:
- op: add
path: /layered_runtime/layers/0/static_layer/circuit_breakers.ai-gateway-extproc-uds.default.max_requests
value: 16384
```

This passes bootstrap validation and yields:

```yaml
layered_runtime:
layers:
- name: global_config
static_layer:
circuit_breakers.ai-gateway-extproc-uds.default.max_requests: 16384
re2.max_program_size.error_level: 4294967295
re2.max_program_size.warn_level: 1000
```

The workaround requires knowing the extension's internal cluster name, hand-writing a JSON pointer against the bootstrap, and restarting proxies to change the value. It is enough to unblock users now, not a substitute for the field.

*Relevant Links*:

- #2051 — removed the `rtds_layer` from the default bootstrap
- #2023 — the RTDS initial-fetch-timeout warning that motivated the removal

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with internal/xds/runner/runner.go and trace the existing bootstrap generation and xDS translation paths, along with the snapshot cache's RuntimeType handling. Add the EnvoyProxy runtime field and ensure the default bootstrap has an RTDS layer and snapshots always include a Runtime resource, including when the map is empty. Verify that editing the field updates RTDS without a proxy restart or CDS churn.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
api, backend-api-design, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.