Azure / Azure/AKS

[Feature] Expose Hubble queue/ring buffer tuning for hubble_lost_events_total in AKS managed Cilium / ACNS

Open
#5,788 2 comments 2 reactions 2 assignees Claimed by @sf-msft View on GitHub
Cilium feature-request
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
4d 15h
Merged PRs (30d)
14

Description

**Is your feature request related to a problem? Please describe.**

Yes. In AKS managed Cilium / Azure CNI Powered by Cilium with ACNS enabled, Hubble can drop observability events during traffic bursts, but users do not appear to have a supported way to tune the Hubble queue or ring buffer.

We observed elevated `hubble_lost_events_total`, mostly from:

- `source="hubble_ring_buffer"`
- `source="observer_events_queue"`

Cilium also logged:

```text
hubble events queue is full: dropping messages; consider increasing the queue size (hubble-event-queue-size) or provisioning more CPU
```

In upstream/self-managed Cilium, the recommended mitigation is to tune settings such as `hubble-event-queue-size` and `hubble-event-buffer-capacity`. In AKS managed Cilium, Cilium configuration is managed by AKS, and manual changes to `cilium-config` are unsupported and may be reverted.

This creates an observability gap: the dataplane may be healthy, but Hubble flow visibility becomes incomplete during high-flow periods.

**Describe the solution you'd like**

Expose a supported AKS / ACNS configuration option for Hubble queue and buffer sizing in Azure CNI Powered by Cilium.

Ideally, AKS should support one or more of the following:

- Cluster-level tuning for values equivalent to `hubble-event-queue-size` and `hubble-event-buffer-capacity`.
- Per-node-pool tuning, because high-flow workloads may be isolated to specific node pools.
- Autosizing based on node size, CPU, memory, or observed flow volume.
- Documented tuning profiles such as `default`, `high-flow`, or `observability-heavy`.
- Documentation explaining how users should respond to `hubble_lost_events_total`.

The configuration should persist across AKS reconciliation and upgrades.

**Describe alternatives you've considered**

- Manually editing `cilium-config`: not suitable because AKS manages Cilium configuration and most fields are unsupported for modification.
- Recycling nodes or Cilium pods: may temporarily reduce symptoms but does not address sustained high-flow observability loss.
- Reducing workload density or moving high-flow workloads to larger nodes: possible mitigation, but indirect and not always practical.
- BYO CNI with self-managed Cilium: provides full control, but is a major operational tradeoff for users who selected AKS managed Cilium.

**Additional context**

Observed in an AKS staging cluster after migrating from Calico to Azure CNI Powered by Cilium.

Environment:

- AKS with Azure CNI Powered by Cilium
- ACNS / Hubble enabled
- Cilium version: `1.18.9`
- Cilium revision: `976ac9d5ca`
- Kubernetes version: `v1.35.1`
- Node OS: `Ubuntu 24.04.4 LTS`
- Kernel: `6.8.0-1052-azure`
- Container runtime: `containerd://2.1.6-2`
- Cilium datapath config: `veth`
- Cilium datapath network mode: `direct-routing`

Observed Hubble-related Cilium startup/config values:

```text
--hubble-event-buffer-capacity='4095'
--hubble-event-queue-size='0'
Configuring Hubble server maxFlows=4095 eventQueueSize=2048
```

Example query used to detect the issue:

```promql
sum by (node, source) (increase(hubble_lost_events_total[24h]))
```

In our case, core dataplane indicators were otherwise healthy:

- No broad container network errors
- No Cilium failing controllers
- No failed endpoint regenerations
- No meaningful policy-denied drops
- No clear workload-wide connectivity symptoms

So this appears to be Hubble observability loss rather than dataplane packet loss. A supported tuning path would help users keep network observability reliable without moving away from AKS managed Cilium.

Image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.