bottlerocket-os / bottlerocket-os/bottlerocket
Add settings.kubernetes.feature-gates to configure kubelet feature gates
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 11
Description
## Summary
Add a `settings.kubernetes.feature-gates` setting that allows users to enable or disable individual kubelet feature gates via Bottlerocket's settings API.
## Motivation
Bottlerocket currently has no mechanism to configure kubelet feature gates. On AL2023, users can patch `/etc/kubernetes/kubelet/kubelet-config.json` directly in `userData` bash scripts, but Bottlerocket's immutable filesystem prevents this approach.
A concrete driving use case: the [AWS Data on EKS best practices guide](https://awslabs.github.io/data-on-eks/docs/bestpractices/analytics/spark-oom-kills) recommends enabling `MemoryQoS=true` on Kubernetes analytics nodes to activate cgroup v2 `memory.high` throttling and reduce OOM kills on memory-intensive workloads (e.g. Apache Spark). Bottlerocket already runs with cgroup v2 enabled — the only missing piece is the kubelet feature gate, which remains alpha and must be explicitly set.
This creates an inconsistency: an AWS-published best practices document recommends a setting that cannot be applied on Bottlerocket, another AWS project, without building a custom image.
Related issues requesting similar or overlapping functionality:
- #1702 — kubelet: add credential provider feature-gate (open since 2021)
- #2481 — Configure kubelet parameters via BR API
- #4056 — Need API setting to allow modify kubelet config
## Proposed API
```toml
[settings.kubernetes.feature-gates]
MemoryQoS = true
```
Individual gates stored as prefixed sub-keys (one key per gate), following the same pattern used by `settings.kubernetes.eviction-hard`. The value type would be `bool`, matching `KubeletConfiguration.featureGates` (`map[string]bool` in the Kubernetes API).
## Implementation scope
This spans three repositories:
| Repo | Change |
|---|---|
| `bottlerocket-os/bottlerocket` | Datastore migration (`AddPrefixesMigration`) |
| `bottlerocket-os/bottlerocket-settings-sdk` | Add `feature_gates: Option>` to `KubernetesSettingsV1` |
| `bottlerocket-os/bottlerocket-core-kit` | Render `featureGates` block in the `kubelet-config` template |
## Open questions
- Is `bool` the right value type, or should it be `String` for forward compatibility?
- Should we validate gate names against a known allowlist, or accept arbitrary strings (unbounded map)?
- Is there a preferred way to coordinate changes across the three repos?
Contributor guide
Assessment
This issue has not been assessed yet.