dynamic modules: add xDS config validator support
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 430
Description
*Description*:
Add a dynamic-modules implementation of Envoy’s existing xDS `ConfigValidator` extension point.
The intent is to let operators enforce deployment-specific safety invariants after Envoy decodes an xDS update but before accepting it—without maintaining an in-tree C++ validator or rebuilding Envoy for each policy.
A representative use case is rejecting a CDS update that removes a required cluster. For ordinary xDS responses, rejection should produce a normal NACK and retain the last accepted configuration.
## Motivation
Control-plane validation remains the first line of defense. An Envoy-side validator adds an independent guard against generator bugs and unsafe control-plane rollouts, operating on the resources Envoy actually decoded.
Today, custom validators must be compiled into Envoy. Dynamic modules already provide a trusted, in-process mechanism for shipping deployment-specific logic independently of the Envoy binary.
This extension runs only on the configuration-update path. It adds no request-path work.
## Proposed configuration
```yaml
config_validators:
- name: envoy.config.validators.dynamic_modules
typed_config:
"@type": type.googleapis.com/envoy.extensions.config.validators.dynamic_modules.v3.DynamicModuleConfigValidator
dynamic_module_config:
name: my_validators
extension_name: required_clusters
extension_config:
"@type": type.googleapis.com/google.protobuf.StringValue
value: cluster_0
type_url: type.googleapis.com/envoy.config.cluster.v3.Cluster
```
Each validator instance selects:
- A module loaded through Envoy’s standard dynamic-module loader.
- An implementation name and opaque extension configuration.
- One canonical xDS resource type URL to validate.
## Proposed ABI
The language-neutral C ABI would provide:
- Config creation and destruction hooks.
- A State-of-the-World validation hook.
- A delta validation hook.
- A callback for attaching a rejection reason.
Decoded resources would cross the ABI as callback-scoped borrowed data containing:
- Resource name.
- Resource version.
- Whether the resource envelope contains a payload.
- Serialized protobuf bytes when a payload is present.
Preserving payload presence distinguishes an absent resource from a present protobuf whose serialization is empty. Delta validation would also receive removed resource names.
Returning success accepts the update. Returning failure, optionally with a reason, rejects an ordinary xDS response through the existing NACK path.
The initial high-level SDK integration would be Rust. Its FFI entry points would contain panics and translate them into configuration rejection rather than allowing unwinding across the C ABI.
## Lifecycle and execution
- Module loading and validation callbacks run synchronously on Envoy’s main thread.
- Buffers are borrowed only for the callback duration.
- The validator configuration owns the loaded module until its destroy callback completes.
- Validators must remain bounded and nonblocking.
- Modules are fully trusted, as with all dynamic modules.
- The initial ABI exposes the update resources, not `Server::Instance` or arbitrary Envoy state.
## Required core API change
`ConfigValidatorFactory::typeUrl()` currently returns one fixed resource type for each registered factory. A generic dynamic-module factory needs the type URL to come from its typed configuration.
The proposed core change is a config-aware overload equivalent to:
```cpp
typeUrl(const Protobuf::Any&, ProtobufMessage::ValidationVisitor&)
```
Existing factories would retain compatibility through a default implementation delegating to `typeUrl()`.
Per `EXTENSION_POLICY.md`, I propose splitting this core extension-point change from the dynamic-module extension PR.
## Initial limitations
- Local filename and by-name module loading only. Remote modules require asynchronous factory context that config validators do not currently receive.
- No shared dynamic-module load/init counters because the validator factory receives no factory context.
- One target xDS type per configured validator.
- Rust receives the first high-level SDK wrapper; other languages can use the C ABI or add wrappers later.
- Proposed extension status is `alpha`.
## TTL expiry question
The existing validator framework can invoke delta validation for TTL expiry from a timer path. There is no discovery response to NACK, and exceptions do not pass through the normal response rejection handler.
Before landing, we should decide whether to:
1. Harden the TTL expiry path with defined rejection semantics; or
2. Define TTL expirations as non-rejectable for config validators.
My preference is to resolve or explicitly constrain this behavior rather than imply ordinary NACK semantics for TTL expiry.
## Alternatives considered
- An in-tree C++ validator: suitable for general Envoy behavior, but requires rebuilding Envoy for deployment-specific policy.
- Control-plane-only validation: useful, but shares the control plane’s failure domain.
- Lua, Wasm, or request filters: these do not participate in the xDS pre-acceptance validation path.
## Validation plan
The implementation would include:
- Module loading, configuration, lifecycle, and canonical type-URL tests.
- State-of-the-World and delta resource serialization tests.
- Absent-versus-empty payload coverage.
- Rejection-message and Rust panic-containment tests.
- End-to-end SOTW, delta, and unified-mux tests proving rejected updates are NACKed and the last accepted state remains active.
- Positive coverage proving accepted updates still apply normally.
A proof of concept is available for design review:
- https://github.com/ElvinEfendi/envoy/tree/dynamic-module-config-validators
- https://github.com/ElvinEfendi/envoy/pull/5
## Sponsorship and ownership
Per the extension policy, this needs a maintainer sponsor and two reviewers.
Would @agrawroh or @mathetake be willing to sponsor? Proposed review coverage, subject to agreement:
- Dynamic modules: @agrawroh, @mathetake, or @wbpcode
- Config validation/xDS: @adisuissa or @yanavlasov
I will hold the upstream implementation PR until the extension shape, TTL semantics, sponsorship, and reviewers are agreed.
Contributor guide
Assessment
This issue has not been assessed yet.