randomparity / randomparity/kdive
Hosted systemd worker proof never runs pytest: third #2054 stdin-drain instance, step reports success
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 26m
- Merged PRs (30d)
- 311
Description
## Problem
The hosted step "Prove systemd worker lifecycle against disposable Postgres" feeds its body to
`bash -s` over a heredoc. `stack-services.sh` (or a child) drains stdin, swallowing the rest of the
script — including the `pytest` invocation — so bash reaches EOF and exits 0.
**The step reports success, and the proof never runs.** All six cases in
`tests/live_vm/test_systemd_worker_lifecycle.py` have never executed in CI.
This is the third instance of **#2054**. The other two were found and fixed; this one was missed.
## Evidence
The step body (`.github/workflows/live.yml`):
```yaml
exec sudo --preserve-env="$preserve_names" \
--user="$operator_name" --group=kdive-live-control \
env "HOME=$HOME" "USER=$operator_name" "LOGNAME=$operator_name" \
/bin/bash -s -e -u -o pipefail \
<<'KDIVE_SYSTEMD_PROOF'
...
scripts/live-stack/stack-services.sh --reset-db --skip-obs --skip-libvirt
source scripts/live-stack/env.sh
KDIVE_RUN_SYSTEMD_WORKER_PROOF=1 uv run python -m pytest \
tests/live_vm/test_systemd_worker_lifecycle.py \
-m live_vm --strict-markers -q
KDIVE_SYSTEMD_PROOF
```
In the most recent scheduled run, that step concluded **success**. Its log ends here:
```
=== next: fund a project ===
The stack is up but no project is funded yet. Seed budget/quota + mint a token with:
just onboard # project 'demo' (override with KDIVE_PROJECT)
##[group]Run sg kdive-live-control -c 'scripts/live-stack/stack-down.sh' <- next step
```
Across the whole ~2900-line step, the only occurrence of the string `pytest` is the echoed
command line. There is no test-session header, no progress output, and **no summary line** —
`-q` always prints one. The step then exits 0 and the job proceeds.
## This exact failure is already documented in the same file, twice
`.github/workflows/live.yml:453`, on the tcg spine:
> \#2054: this run block reaches bash over stdin, and a script read from stdin is consumed
> incrementally — stack-services.sh's libvirt-provisioning leg once ran a child that drained stdin
> to EOF, swallowing every not-yet-read spine byte, **so the tier exited 0 right after
> stack-services.sh's banner with onboard/preflight/pytest never reached.** Materialize the body to
> a file and execute THAT file, removing the class outright.
`.github/workflows/live.yml:712`, on the native spine, says the same and applies the same fix.
The observed symptom above matches that description word for word: exit 0, right after
`stack-services.sh`'s banner, with pytest never reached. Both fixed steps now do
`cat >"$RUNNER_TEMP/spine-*.sh" <<'...'` and execute the file. This step still pipes into
`bash -s`.
## Why it matters
The six cases that never ran are the only proof of the fixed systemd worker lifecycle contract
(ADR-0574):
- `test_real_systemd_workers_register_heartbeat_and_terminate[1]`
- `test_real_systemd_workers_register_heartbeat_and_terminate[3]`
- `test_database_outage_retains_exact_invocation_until_stop_retry`
- `test_recover_clears_the_failed_identity_that_blocks_the_next_start`
- `test_recover_retires_a_restarted_slot_and_releases_its_fence_in_one_call`
- `test_recover_refuses_a_live_slot_without_releasing_its_fence`
The last three are added by PR #2542. **A green check on that PR therefore says nothing about
them**, which is the specific risk of a gate that cannot fail.
Run for the first time on a real provisioned host, **all six fail** — see the companion issue. So
this is not a hypothetical gap: the gate has been green over failing tests.
## Expected
The step materializes its body to a file and executes that file, exactly as the two sibling steps
already do, so no child process can consume unread script bytes.
Worth considering alongside it: this class is invisible precisely because the truncated script
exits 0. The tcg spine already guards against the sibling failure with an explicit
`grep -Eq '(^|[[:space:],])[1-9][0-9]* passed'` check on its summary, with the rationale that
"a skipped tier must never read green". The same assertion here would have caught this on the
first run, and would catch any future regression of the same class regardless of cause.
## Out of scope
- The six test failures themselves — companion issue; they are what this gate was supposed to
surface.
- The two already-fixed `#2054` instances at lines 453 and 712.
## Provenance
Found while running these tests on a provisioned host to obtain a live proof for PR #2542. The
tests failed there, which prompted checking what CI reports for the same file; CI reported success
with no pytest output at all. The step's conclusion, the absence of any `-q` summary in its log,
and the two pre-existing `#2054` comments describing this precise symptom were each read directly
rather than inferred.
Contributor guide
Research direction
Start in .github/workflows/live.yml at the hosted “Prove systemd worker lifecycle against disposable Postgres” step, then compare the fixed tcg and native spine steps around lines 453 and 712. Check tests/live_vm/test_systemd_worker_lifecycle.py and the logged command first. Done means the step executes the complete materialized script and verifies a pytest summary before reporting success.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, github-actions, python
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100