bottlerocket-os / bottlerocket-os/bottlerocket
eliminate use of `systemctl try-restart`
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 12
Description
Today we use `systemctl try-restart` to attempt a service restart after applying settings. Partly this is because we process settings early in the boot, when the affected services haven't been started yet and aren't intended to start.
However, this causes trouble when changing settings at runtime, because if the service isn't running, the command will do nothing.
Services might not be running for a few reasons:
* they failed to start after bad settings were previously applied
* they are starting after new settings are applied, but aren't yet started all the way
In a host container running at boot, @vignesh-goutham discovered the following race:
* host container queries systemd for the status of `kubelet`
* waits for it to finish activating (`ActiveState=active` and `SubState=running`)
* issues `apiclient set` commands to reconfigure `kubelet`
* `apiserver` executes restart commands
* `systemctl try-restart` does nothing
* `systemd` logs the first `Started Kubelet` around 1 second later
From this we can infer that two calls to `apiclient set kubernetes.` in quick succession will not always result in two `kubelet` restarts, leaving that service in an undefined state.
For changing settings at runtime, we really need something more like `force-stop` and `force-start` to ensure that the restart commands are fully enacted for each transaction.
Contributor guide
Research direction
Start by tracing how the apiserver handles `apiclient set` transactions and where it invokes `systemctl try-restart`; review the early-boot and runtime cases described in the issue. Verify the restart behavior when a service is stopped, failed, or still activating, and ensure each transaction's restart is fully enacted without breaking early-boot handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100