randomparity / randomparity/kdive
Every systemd worker lifecycle case fails on a real host: control-group assertion ignores the per-template slice
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 26m
- Merged PRs (30d)
- 311
Description
## Problem
Run against a provisioned host, every case in `tests/live_vm/test_systemd_worker_lifecycle.py`
fails:
```
FAILED test_real_systemd_workers_register_heartbeat_and_terminate[1]
FAILED test_real_systemd_workers_register_heartbeat_and_terminate[3]
FAILED test_database_outage_retains_exact_invocation_until_stop_retry
FAILED test_recover_clears_the_failed_identity_that_blocks_the_next_start
FAILED test_recover_retires_a_restarted_slot_and_releases_its_fence_in_one_call
FAILED test_recover_refuses_a_live_slot_without_releasing_its_fence
============================== 6 failed in 31.24s ==============================
```
There is **one** root failure; the other five are a cascade from it.
This went unnoticed because the CI step that runs this file never actually invokes pytest — see
the companion issue. These tests have not executed anywhere.
## Root cause
`_assert_started` requires a worker unit's control group to sit directly under `system.slice`:
```python
# tests/live_vm/test_systemd_worker_lifecycle.py:271
assert unit.control_group == f"/system.slice/{unit.unit}"
```
```
E AssertionError
E - /system.slice/kdive-live-worker@1.service
E + /system.slice/system-kdive\x2dlive\x2dworker.slice/kdive-live-worker@1.service
```
systemd places instances of a template unit in an automatically-created per-template slice.
`kdive-live-worker@.service` declares no `Slice=`, so on the host under test
(systemd 259) the instances land in `system-kdive\x2dlive\x2dworker.slice`, which systemd
created and lists as loaded and active:
```
system-kdive\x2dlive\x2dworker.slice loaded active active Slice /system/kdive-live-worker
system-kdive\x2dlive\x2dworker\x2dlifecycle.slice loaded active active Slice /system/kdive-live-worker-lifecycle
```
The assertion therefore cannot hold for a templated unit on a current systemd. Either the
expectation should account for the per-template slice, or the units should declare an explicit
`Slice=` if placement directly under `system.slice` is a real requirement rather than an
assumption — that is the decision this issue should settle, since the unit files and the
assertion currently disagree about which is intended.
## The cascade
`test_real_systemd_workers_register_heartbeat_and_terminate[1]` fails the assertion above and
aborts without stopping the fleet it started. Every later case then fails on its own
precondition:
```
subprocess.CalledProcessError: Command
('scripts/live-stack/worker-lifecycle.sh', 'start', '1') returned non-zero exit status 4
```
because the fleet is left in:
```json
{"ok": false, "code": "conflict",
"message": "systemd observation conflicts with the retained lifecycle contract",
"retry_action": "operator_recovery",
"slots": [{"slot": 1, "phase": "started", "code": "conflict"}, ...]}
```
So the three `recover` cases never reach the behaviour they are written to exercise — they fail
in `_assert_started`, before any `recover` call. **Nothing here is evidence for or against the
`recover` operation itself.**
Worth fixing alongside: a case that starts a fleet and then fails an assertion leaves the host
unusable for every subsequent case. Tearing the fleet down in a fixture teardown rather than
inline would turn this from six failures into one.
## An observation, deliberately not a claim
With the fleet in the conflicted state above, invoking the new operation directly returns
`conflict` and leaves the state unchanged:
```
$ worker-lifecycle.sh recover → exit 4, code "conflict", slots 1-3 "conflict"
$ worker-lifecycle.sh status → unchanged
```
At that moment the units are `active` with `SubState=exited` and an empty `ControlGroup` — the
processes are gone while systemd still reports the units active.
This may be correct: `test_recover_refuses_a_live_slot_without_releasing_its_fence` exists
precisely so recovery never retires a running worker, and refusing anything systemd calls
`active` is the conservative reading. It may also be a gap, if `active (exited)` with no
processes is meant to be recoverable. **Deciding that requires the tests to run, which is what
this issue and its companion block.** It is recorded here so the question is not lost, not as a
defect report.
## Expected
The file passes on a provisioned host, and the `recover` cases reach the behaviour they name.
## Out of scope
- The CI step that never runs this file — companion issue. That is what allowed these to rot.
- PR #2542's non-test changes. Its provisioning half was separately verified on a real host: the
installed lifecycle identity moved as intended on reprovision, and its new `verify.yml`
assertion ran and passed.
## Provenance
Found by running the file on a host provisioned from scratch for a live proof of PR #2542. The
root assertion, the cascade, the resulting fleet state, the systemd slice listing and the unit
sub-states were each read directly from the host. The distinction between the one real failure and
the five consequential ones was established from the tracebacks, not assumed from the count.
Contributor guide
Research direction
Start with tests/live_vm/test_systemd_worker_lifecycle.py, especially _assert_started around line 271, and run the file on a provisioned host. Compare the assertion with the per-template systemd slice shown in the failure, then settle whether placement or the expectation is intended. Ensure the file passes and the recover cases reach the behavior they name; consider fixture teardown so a failed assertion does not leave the fleet running.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100