[Feature] Expose Hubble queue/ring buffer tuning for hubble_lost_events_total in AKS managed Cilium / ACNS
- Dominant language
- TypeScript
- Stars
- 2.1k
- Forks
- 395
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 14
Description
**Is your feature request related to a problem? Please describe.**
Yes. In AKS managed Cilium / Azure CNI Powered by Cilium with ACNS enabled, Hubble can drop observability events during traffic bursts, but users do not appear to have a supported way to tune the Hubble queue or ring buffer.
We observed elevated `hubble_lost_events_total`, mostly from:
- `source="hubble_ring_buffer"`
- `source="observer_events_queue"`
Cilium also logged:
```text
hubble events queue is full: dropping messages; consider increasing the queue size (hubble-event-queue-size) or provisioning more CPU
```
In upstream/self-managed Cilium, the recommended mitigation is to tune settings such as `hubble-event-queue-size` and `hubble-event-buffer-capacity`. In AKS managed Cilium, Cilium configuration is managed by AKS, and manual changes to `cilium-config` are unsupported and may be reverted.
This creates an observability gap: the dataplane may be healthy, but Hubble flow visibility becomes incomplete during high-flow periods.
**Describe the solution you'd like**
Expose a supported AKS / ACNS configuration option for Hubble queue and buffer sizing in Azure CNI Powered by Cilium.
Ideally, AKS should support one or more of the following:
- Cluster-level tuning for values equivalent to `hubble-event-queue-size` and `hubble-event-buffer-capacity`.
- Per-node-pool tuning, because high-flow workloads may be isolated to specific node pools.
- Autosizing based on node size, CPU, memory, or observed flow volume.
- Documented tuning profiles such as `default`, `high-flow`, or `observability-heavy`.
- Documentation explaining how users should respond to `hubble_lost_events_total`.
The configuration should persist across AKS reconciliation and upgrades.
**Describe alternatives you've considered**
- Manually editing `cilium-config`: not suitable because AKS manages Cilium configuration and most fields are unsupported for modification.
- Recycling nodes or Cilium pods: may temporarily reduce symptoms but does not address sustained high-flow observability loss.
- Reducing workload density or moving high-flow workloads to larger nodes: possible mitigation, but indirect and not always practical.
- BYO CNI with self-managed Cilium: provides full control, but is a major operational tradeoff for users who selected AKS managed Cilium.
**Additional context**
Observed in an AKS staging cluster after migrating from Calico to Azure CNI Powered by Cilium.
Environment:
- AKS with Azure CNI Powered by Cilium
- ACNS / Hubble enabled
- Cilium version: `1.18.9`
- Cilium revision: `976ac9d5ca`
- Kubernetes version: `v1.35.1`
- Node OS: `Ubuntu 24.04.4 LTS`
- Kernel: `6.8.0-1052-azure`
- Container runtime: `containerd://2.1.6-2`
- Cilium datapath config: `veth`
- Cilium datapath network mode: `direct-routing`
Observed Hubble-related Cilium startup/config values:
```text
--hubble-event-buffer-capacity='4095'
--hubble-event-queue-size='0'
Configuring Hubble server maxFlows=4095 eventQueueSize=2048
```
Example query used to detect the issue:
```promql
sum by (node, source) (increase(hubble_lost_events_total[24h]))
```
In our case, core dataplane indicators were otherwise healthy:
- No broad container network errors
- No Cilium failing controllers
- No failed endpoint regenerations
- No meaningful policy-denied drops
- No clear workload-wide connectivity symptoms
So this appears to be Hubble observability loss rather than dataplane packet loss. A supported tuning path would help users keep network observability reliable without moving away from AKS managed Cilium.
Contributor guide
Assessment
This issue has not been assessed yet.