dynamic_modules: add configurable shared stats scopes and cardinality limits
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 430
Description
## Summary
Add optional `envoy.type.v3.Scope stats_scope` configuration to the common
`DynamicModuleConfig`. Apply it to every implemented dynamic-module metric producer.
This provides opt-in counter, gauge, and histogram cardinality limits, including a
shared process-wide budget across multiple configurations. No dynamic-module ABI or SDK
change is required.
This bounds retained Envoy metric-series count. It does not bound arbitrary module
memory, metric-name bytes, or transient per-callback allocation.
## Motivation
Dynamic modules can create request-dependent metric series:
- #41002 requested metrics with runtime labels.
- #41003 introduced metric vectors with fixed label names and runtime label values.
- Each admitted label-value tuple creates a concrete Envoy stat.
- Modules can also define many fixed metrics during configuration.
Dynamic-module scopes currently use unlimited stats cardinality by default. Envoy has
opt-in per-scope limits from #42168, but dynamic modules cannot configure them.
#45316 demonstrates the operational need for bounded high-cardinality stats, although
it was not a dynamic-module incident.
## Proposed API
```proto
message DynamicModuleConfig {
// Existing fields omitted.
// Optional scope configuration for metrics emitted by this dynamic module.
//
// Dynamic modules do not currently support stat eviction.
envoy.type.v3.Scope stats_scope = 8;
}
```
Example:
```yaml
dynamic_module_config:
name: example
stats_scope:
prefix: example_module
sharing_name: example_module_metrics
max_counters: 1000
max_gauges: 500
max_histograms: 100
```
Prefix compatibility:
- Reject configurations where both `metrics_namespace` and `stats_scope.prefix` are
non-empty.
- Otherwise use `stats_scope.prefix`, then `metrics_namespace`, then the existing
extension-specific default.
- Allow `metrics_namespace` together with limits-only `stats_scope`.
- Do not deprecate `metrics_namespace` in this change.
- Preserve existing names and unlimited behavior when `stats_scope` is absent.
## Scope ownership and sharing
Apply limits directly to the final scope used by metric callbacks. Limits on a parent
scope do not propagate into subsequently created child scopes.
- Empty `sharing_name`: create a distinct scope under the extension's existing parent.
- Non-empty `sharing_name`: use a process-wide shared scope rooted at the server scope.
This re-rooting can affect metric names and must be documented.
- Shared identity includes a dynamic-module consumer domain and the hash of the complete
effective `Scope` configuration.
- All dynamic-module extension points use the same domain, allowing intentional sharing
between them.
- Stats Access Logger uses a separate domain, preventing accidental cross-feature
sharing.
Generalize the scope provider introduced by #43625 rather than creating a second
sharing implementation.
## Limit behavior
- Limits apply independently to counters, gauges, and histograms.
- An absent limit is unlimited.
- Explicit zero rejects every new stat of that type.
- Fixed metrics and instantiated vector label tuples consume the same per-type budget.
- Existing stats continue updating after the limit is reached.
- A rejected creation returns Envoy's no-op stat.
- The existing ABI continues returning `Success`.
- `server.stats_overflow.{counter,gauge,histogram}` increments for every rejected
lookup or creation attempt. These stats do not represent dropped values or unique
rejected series.
- Dynamic modules currently expose no text-readout producer API.
No request-path warning logs are proposed; overflow counters provide bounded
observability.
## Eviction
Reject `enable_eviction: true` during dynamic-module configuration validation.
Several implementations retain raw counter, gauge, or histogram references. Eviction
can make those references stale while the ABI handle remains live. #44710 and #43812
demonstrate analogous lifecycle hazards.
Eviction support requires a later handle redesign that stores metric identity and
re-resolves through the scope.
## Producer coverage
Wire all ten implemented metric producers:
1. HTTP filter
2. Network filter
3. Listener filter
4. UDP listener filter
5. Dynamic-module access logger
6. Bootstrap extension
7. Cluster
8. Load-balancing policy
9. Tracer
10. Stats sink
DNS resolver metric callbacks are excluded. They currently have ABI/SDK declarations
but only weak unimplemented Envoy stubs.
When `stats_scope` is absent, preserve existing scope construction exactly. Cluster and
stats sink should only enter the new path when `stats_scope` is configured, avoiding
unrelated namespace changes.
## Implementation stages
1. Fix HTTP stream metric callbacks to use callback-local `StatNameDynamicPool`
instances. #44840 established this pattern in other extension points; HTTP streams
still retain every label value for the stream lifetime.
2. Add the API, generalize the shared-scope provider, and integrate HTTP.
3. Wire the remaining nine implemented producer points.
## Validation
Use `ThreadLocalStoreImpl`; `IsolatedStoreImpl` does not enforce these limits.
Cover:
- Fixed metrics and vector label tuples.
- Counters, gauges, and histograms.
- Explicit zero and exact-boundary behavior.
- Existing-stat updates after overflow.
- Overflow counter semantics.
- Shared aggregate budgets across configurations.
- Independent unshared scopes.
- Cross-consumer-domain isolation.
- Prefix compatibility and legacy metric names.
- Eviction rejection before module initialization.
- HTTP label-pool retained-memory regression.
- Mutation probes proving the limit and shared-scope tests fail when enforcement is
bypassed.
## Non-goals
- A default finite limit.
- Process-wide limits for unrelated Envoy stats.
- Eviction-safe ABI handles.
- A new ABI overflow result.
- Limits on metric definitions, label count, or label bytes.
- Fixing unrelated legacy namespace behavior.
## Related work
- #40395 - metric eviction
- #42168 - per-scope stat limits
- #43625 - generic `Scope` configuration and shared-scope precedent
- #41002 / #41003 - dynamic-module metrics with labels
- #44840 - callback-local metric label pools
- #44710 / #43812 - eviction and retained-reference hazards
- #45316 - operational high-cardinality backport
- #45765 - proposed access-logger metrics with labels
Seeking agreement from dynamic-modules maintainers, stats maintainers, and API
shepherds before implementation.
Contributor guide
Assessment
This issue has not been assessed yet.