envoyproxy / envoyproxy/envoy

dynamic modules: add xDS config validator support

Open
#46,470 0 comments 0 reactions 1 assignee Claimed by @agrawroh View on GitHub
area/dynamic_modules enhancement no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

*Description*:

Add a dynamic-modules implementation of Envoy’s existing xDS `ConfigValidator` extension point.

The intent is to let operators enforce deployment-specific safety invariants after Envoy decodes an xDS update but before accepting it—without maintaining an in-tree C++ validator or rebuilding Envoy for each policy.

A representative use case is rejecting a CDS update that removes a required cluster. For ordinary xDS responses, rejection should produce a normal NACK and retain the last accepted configuration.

## Motivation

Control-plane validation remains the first line of defense. An Envoy-side validator adds an independent guard against generator bugs and unsafe control-plane rollouts, operating on the resources Envoy actually decoded.

Today, custom validators must be compiled into Envoy. Dynamic modules already provide a trusted, in-process mechanism for shipping deployment-specific logic independently of the Envoy binary.

This extension runs only on the configuration-update path. It adds no request-path work.

## Proposed configuration

```yaml
config_validators:
- name: envoy.config.validators.dynamic_modules
typed_config:
"@type": type.googleapis.com/envoy.extensions.config.validators.dynamic_modules.v3.DynamicModuleConfigValidator
dynamic_module_config:
name: my_validators
extension_name: required_clusters
extension_config:
"@type": type.googleapis.com/google.protobuf.StringValue
value: cluster_0
type_url: type.googleapis.com/envoy.config.cluster.v3.Cluster
```

Each validator instance selects:

- A module loaded through Envoy’s standard dynamic-module loader.
- An implementation name and opaque extension configuration.
- One canonical xDS resource type URL to validate.

## Proposed ABI

The language-neutral C ABI would provide:

- Config creation and destruction hooks.
- A State-of-the-World validation hook.
- A delta validation hook.
- A callback for attaching a rejection reason.

Decoded resources would cross the ABI as callback-scoped borrowed data containing:

- Resource name.
- Resource version.
- Whether the resource envelope contains a payload.
- Serialized protobuf bytes when a payload is present.

Preserving payload presence distinguishes an absent resource from a present protobuf whose serialization is empty. Delta validation would also receive removed resource names.

Returning success accepts the update. Returning failure, optionally with a reason, rejects an ordinary xDS response through the existing NACK path.

The initial high-level SDK integration would be Rust. Its FFI entry points would contain panics and translate them into configuration rejection rather than allowing unwinding across the C ABI.

## Lifecycle and execution

- Module loading and validation callbacks run synchronously on Envoy’s main thread.
- Buffers are borrowed only for the callback duration.
- The validator configuration owns the loaded module until its destroy callback completes.
- Validators must remain bounded and nonblocking.
- Modules are fully trusted, as with all dynamic modules.
- The initial ABI exposes the update resources, not `Server::Instance` or arbitrary Envoy state.

## Required core API change

`ConfigValidatorFactory::typeUrl()` currently returns one fixed resource type for each registered factory. A generic dynamic-module factory needs the type URL to come from its typed configuration.

The proposed core change is a config-aware overload equivalent to:

```cpp
typeUrl(const Protobuf::Any&, ProtobufMessage::ValidationVisitor&)
```

Existing factories would retain compatibility through a default implementation delegating to `typeUrl()`.

Per `EXTENSION_POLICY.md`, I propose splitting this core extension-point change from the dynamic-module extension PR.

## Initial limitations

- Local filename and by-name module loading only. Remote modules require asynchronous factory context that config validators do not currently receive.
- No shared dynamic-module load/init counters because the validator factory receives no factory context.
- One target xDS type per configured validator.
- Rust receives the first high-level SDK wrapper; other languages can use the C ABI or add wrappers later.
- Proposed extension status is `alpha`.

## TTL expiry question

The existing validator framework can invoke delta validation for TTL expiry from a timer path. There is no discovery response to NACK, and exceptions do not pass through the normal response rejection handler.

Before landing, we should decide whether to:

1. Harden the TTL expiry path with defined rejection semantics; or
2. Define TTL expirations as non-rejectable for config validators.

My preference is to resolve or explicitly constrain this behavior rather than imply ordinary NACK semantics for TTL expiry.

## Alternatives considered

- An in-tree C++ validator: suitable for general Envoy behavior, but requires rebuilding Envoy for deployment-specific policy.
- Control-plane-only validation: useful, but shares the control plane’s failure domain.
- Lua, Wasm, or request filters: these do not participate in the xDS pre-acceptance validation path.

## Validation plan

The implementation would include:

- Module loading, configuration, lifecycle, and canonical type-URL tests.
- State-of-the-World and delta resource serialization tests.
- Absent-versus-empty payload coverage.
- Rejection-message and Rust panic-containment tests.
- End-to-end SOTW, delta, and unified-mux tests proving rejected updates are NACKed and the last accepted state remains active.
- Positive coverage proving accepted updates still apply normally.

A proof of concept is available for design review:

- https://github.com/ElvinEfendi/envoy/tree/dynamic-module-config-validators
- https://github.com/ElvinEfendi/envoy/pull/5

## Sponsorship and ownership

Per the extension policy, this needs a maintainer sponsor and two reviewers.

Would @agrawroh or @mathetake be willing to sponsor? Proposed review coverage, subject to agreement:

- Dynamic modules: @agrawroh, @mathetake, or @wbpcode
- Config validation/xDS: @adisuissa or @yanavlasov

I will hold the upstream implementation PR until the extension shape, TTL semantics, sponsorship, and reviewers are agreed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.