bottlerocket-os / bottlerocket-os/bottlerocket

Expose the overlayfs `volatile` mount option for the kubernetes containerd snapshotter

Open
#4,885 0 comments 0 reactions 0 assignees View on GitHub
status/needs-triage type/enhancement
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

## What I'd like

A setting to make containerd mount container rootfs overlays with `volatile`, for example:

```toml
[settings.container-runtime]
overlay-volatile = true
```

Containerd already supports this via its overlay snapshotter config (`mount_options = ['volatile']`), so this is only about exposing it. (Alternatively, let user drop-ins survive in `/etc/containerd/config.d`; today files written there by bootstrap containers are removed by the settings render.)

## Why

With `volatile`, images pull 20% faster and various other things (eg warming caches at boot, or just writing heavily) proceed at their CPU and network limits, and happily use the page cache.

Without `volatile`, unmounting any container rootfs (eg when startup containers run or images are pulled) causes a `syncfs` on the whole backing filesystem. Because EC2 filesystems are often very slow, this causes noticeable slowdowns. The intent is durability to system crashes, yet container rootfs resilience is not something that is important on Kubernetes. I'm struggling to even imagine what the `syncfs` is trying to achieve. Being sure that a container's state has been saved to disk in a happens-before relationship with the stop command return? There's no state. This isn't desktop Docker. The rootfs syncfs doesn't even protect real data, which is typically on other filesystems. Honestly, `volatile` should simply be the default.

## Related

- https://www.redhat.com/en/blog/container-volatile-overlay-mounts
- https://github.com/containerd/containerd/issues/6406
- #2408 (custom containerd configuration via userdata — closed, custom-AMI only)

## Background

From the RedHat link above:

> ### What is syncing, and why is it important?
>
> In Linux, when you write to a file or directory, the kernel does not instantly write the data to disk. Instead, it buffers up a bunch of writes and then periodically saves the data to disk to increase performance. This is called a sync. The problem with this is that a process thinks the data was saved when the write completes, but it really isn't until the kernel syncs that data. This means that if you wrote data and the kernel crashed, there is a chance that the data was never saved.
>
> Because of this, lots of file systems sync regularly, and tools can request syncing to happen often. When a sync occurs, the kernel stops processing data with a lock and syncs all of the data to disk. Of course, this causes poorer performance. If you have a process that causes syncs frequently, your job’s performance can really be hurt. Certain tools like RPM call for a sync after every file is written to disk, causing all the dirty pages for that file to be flushed, and it is a considerable overhead.
>
> ### Containers may not need syncing
>
> In the container world, we have many use cases where we don’t care if the data is saved. If the kernel crashed, we would not use the written data anyway.
>
> ...
>
> In the Kubernetes world, CRI-O is the container engine. Kubernetes is almost always set up to remove all containers at boot time. Basically, it wants to start with a clean state. This means if the kernel crashed while data was being written to the overlay mount, this data would be destroyed as soon as the system boots. It is also safe to use such configurations with stateful containers because the data is usually written to external volumes that won’t be affected by the “volatile” flag at runtime.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the `settings.container-runtime` configuration through the settings render and containerd overlay snapshotter setup. Check how `/etc/containerd/config.d` drop-ins are handled, then verify that enabling `overlay-volatile` results in the requested `mount_options = ['volatile']` behavior without breaking the existing configuration path.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, linux
Domain
operating-systems
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.