bottlerocket-os / bottlerocket-os/bottlerocket

eliminate use of `systemctl try-restart`

Open
#1,711 4 comments 1 reaction 0 assignees View on GitHub
area/core help wanted status/icebox type/bug
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 18h
Merged PRs (30d)
12

Description

Today we use `systemctl try-restart` to attempt a service restart after applying settings. Partly this is because we process settings early in the boot, when the affected services haven't been started yet and aren't intended to start.

However, this causes trouble when changing settings at runtime, because if the service isn't running, the command will do nothing.

Services might not be running for a few reasons:
* they failed to start after bad settings were previously applied
* they are starting after new settings are applied, but aren't yet started all the way

In a host container running at boot, @vignesh-goutham discovered the following race:
* host container queries systemd for the status of `kubelet`
* waits for it to finish activating (`ActiveState=active` and `SubState=running`)
* issues `apiclient set` commands to reconfigure `kubelet`
* `apiserver` executes restart commands
* `systemctl try-restart` does nothing
* `systemd` logs the first `Started Kubelet` around 1 second later

From this we can infer that two calls to `apiclient set kubernetes.` in quick succession will not always result in two `kubelet` restarts, leaving that service in an undefined state.

For changing settings at runtime, we really need something more like `force-stop` and `force-start` to ensure that the restart commands are fully enacted for each transaction.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how the apiserver handles `apiclient set` transactions and where it invokes `systemctl try-restart`; review the early-boot and runtime cases described in the issue. Verify the restart behavior when a service is stopped, failed, or still activating, and ensure each transaction's restart is fully enacted without breaking early-boot handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.