bottlerocket-os / bottlerocket-os/bottlerocket

Add settings.kubernetes.feature-gates to configure kubelet feature gates

Open
#4,814 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

## Summary

Add a `settings.kubernetes.feature-gates` setting that allows users to enable or disable individual kubelet feature gates via Bottlerocket's settings API.

## Motivation

Bottlerocket currently has no mechanism to configure kubelet feature gates. On AL2023, users can patch `/etc/kubernetes/kubelet/kubelet-config.json` directly in `userData` bash scripts, but Bottlerocket's immutable filesystem prevents this approach.

A concrete driving use case: the [AWS Data on EKS best practices guide](https://awslabs.github.io/data-on-eks/docs/bestpractices/analytics/spark-oom-kills) recommends enabling `MemoryQoS=true` on Kubernetes analytics nodes to activate cgroup v2 `memory.high` throttling and reduce OOM kills on memory-intensive workloads (e.g. Apache Spark). Bottlerocket already runs with cgroup v2 enabled — the only missing piece is the kubelet feature gate, which remains alpha and must be explicitly set.

This creates an inconsistency: an AWS-published best practices document recommends a setting that cannot be applied on Bottlerocket, another AWS project, without building a custom image.

Related issues requesting similar or overlapping functionality:
- #1702 — kubelet: add credential provider feature-gate (open since 2021)
- #2481 — Configure kubelet parameters via BR API
- #4056 — Need API setting to allow modify kubelet config

## Proposed API

```toml
[settings.kubernetes.feature-gates]
MemoryQoS = true
```

Individual gates stored as prefixed sub-keys (one key per gate), following the same pattern used by `settings.kubernetes.eviction-hard`. The value type would be `bool`, matching `KubeletConfiguration.featureGates` (`map[string]bool` in the Kubernetes API).

## Implementation scope

This spans three repositories:

| Repo | Change |
|---|---|
| `bottlerocket-os/bottlerocket` | Datastore migration (`AddPrefixesMigration`) |
| `bottlerocket-os/bottlerocket-settings-sdk` | Add `feature_gates: Option>` to `KubernetesSettingsV1` |
| `bottlerocket-os/bottlerocket-core-kit` | Render `featureGates` block in the `kubelet-config` template |

## Open questions

- Is `bool` the right value type, or should it be `String` for forward compatibility?
- Should we validate gate names against a known allowlist, or accept arbitrary strings (unbounded map)?
- Is there a preferred way to coordinate changes across the three repos?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.