bottlerocket-os / bottlerocket-os/bottlerocket

BOTTLEROCKET_PRIVATE can lose/corrupt data on power loss

Open
#3,635 3 comments 0 reactions 0 assignees View on GitHub
area/metal status/needs-proposal type/bug
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

In a very specific situation I have a bunch of devices that are getting power cycled _immediately_ after first boot is complete. In many cases the device then fails to boot a second time. Upon inspection of the BOTTLEROCKET_PRIVATE filesystem of a device that fails to boot the datastore is incomplete/corrupted and fails to boot the second time due to not being able to synthesize a proper system configuration. If I put in a sync/drop caches command in a bootstrap container I notice that the settings never become corrupted.

**Image I'm using:**
Custom **metal-dev** variant

**What I expected to happen:**
The system will boot multiple times in a row even if there is a power loss

**What actually happened:**
The system only boots the first time when there is a power loss immediately after first boot

**How to reproduce the problem:**
This one is a bit tough. I have a screen that is displaying information on an HDMI port from a bootstrap container. Once I see that output I pull power from the system. Boot a second time and notice how the settings are corrupted and the system won't boot.

I'm speculating at this point due to testing with a hard **sync** right before the power loss that this is due to a file write-cache issue. The settings are in flight in the write cache but don't get synced out to disk. Although this is a bit of a corner case does it make sense to mount BOTTLEROCKET_PRIVATE without a write-cache so writes are guaranteed to hit the disk since that partition is a low performance target anyway?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.