Azure / Azure/AKS

[Feature] Cilium request values

Open
#5,244 6 comments 3 reactions 2 assignees Claimed by @sf-msft View on GitHub
Cilium feature-request
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

Azure CNI Managed by Cilium currently lacks request values for the agent pods and cilium-operator pods. The agent pods request 100Mi of memory based on the request value of the init-container clean-cilium-state, while all other containers do not have specified request values. This situation can lead to OOMKilled events on nodes if the request value is too low for the cluster's usage. Without a limit value set for the cilium-agent, it may consume more memory than requested, potentially causing other workloads to be terminated by the node during an OOMKilled event.

In addition to Cilium, other workloads like acns-security also lack request values, leading to resource under-provisioning for system components. This can result in the termination of system or user pods and can also hinder autoscaling mechanisms such as Node Autoprovision or Cluster Autoscaler.

To address these issues, we have temporarily increased the System node pool SKU and minimum count to accommodate the unrequested resources. However, this mitigation is not suitable for User node pools, as user workloads are dynamic and will utilize unrequested resources as needed during operation.

As a second mitigation we are using a LimitRange resource in the kube-system namespace:
```yaml
apiVersion: v1
kind: LimitRange
metadata:
name: defaultresources
namespace: kube-system
spec:
limits:
- default:
memory: 500Mi
defaultRequest:
memory: 500Mi
cpu: 250m
type: Container
```
For a long-term solution, we are requesting that the cilium-agent be assigned appropriate request values. To accommodate the dynamic nature of resource requirements, it would be beneficial to leverage Vertical Pod Autoscaler (VPA) for managing resource values, as outlined in issue #2125. We also propose the inclusion of Cilium in this optimization effort.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.