canonical / canonical/checkbox
stress-ng-test-for-stressor-umount-root can stall the entire test session (leaked/hung stressor process, no enforced job-level timeout)
- Dominant language
- Python
- Stars
- 52
- Forks
- 81
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 69
Description
## Summary
`stress/stress-ng-test-for-stressor-umount-root` (and likely other `stress-ng-test-for-stressor-*` jobs) can leave the DUT/test session unable to continue after the stressor itself has finished, because the job has no enforced mechanism to detect and terminate a hung/leaked process tree. Two independent reports now show this stalling the remainder of a test plan on two different SoCs/kernels:
- This report: Qualcomm X1E80100 (Hamoa IoT EVK), kernel `7.0.0-1013-qcom`, server image — `stress-ng` reports the umount stressor completed successfully, but orphaned `stress-ng-umount` worker processes remain alive (reparented to init), blocked indefinitely in `ppoll()`, evidently still holding the job's output pipe open.
- Related: [LP #2166125](https://bugs.launchpad.net/baoshan/+bug/2166125) — MediaTek Genio 720, kernel `6.8.0-1006-mtk`, desktop image — same job (`stress-ng-test-for-stressor-umount-root`), a different suspected mechanism (D-Bus match-rule quota flooding from rapid mount/umount events), but the same end symptom: "Stressor umount works well, but Checkbox is stopped/killed, which causes the whole testing session to stop there."
Two different platforms, two different plausible underlying causes, same symptom family: the umount stressor job can leave the harness/session unable to proceed to the next job.
## What is proven (this report, Qualcomm platform)
Job command reproduced exactly as defined in
`checkbox-provider-base/units/stress/stress-ng.yaml` (template
`stress-ng-test-for-stressor-{stressor}-root`):
```
cd /var/tmp
stress-ng --umount 0 --timeout 30 --oom-avoid-bytes 10% --skip-silent --verbose
```
(reproduced with `--timeout 15` for faster iteration; behavior identical)
`stress-ng` itself reports full success:
```
umount: [116433] terminated (success)
metrics-check: all stressor metrics validated and sane
passed: 1: umount (1) failed: 0
successful run completed in 15.00 secs
```
But after the parent process exits, child processes remain alive, reparented to init:
```
PID PPID STAT ELAPSED COMMAND
116270 1 S 06:00 stress-ng-umoun
116271 1 S 06:00 stress-ng-umoun
116273 116270 S 06:00 stress-ng-umoun
```
Kernel stack of one such process, still blocked 6 minutes after parent exit:
```
poll_schedule_timeout.constprop.0 -> do_poll.constprop.0 -> do_sys_poll ->
__arm64_sys_ppoll -> invoke_syscall.constprop.0 -> el0_svc_common ...
```
i.e. blocked indefinitely in `ppoll()`. No leftover mount-namespace entries were found (`mount | grep stress-ng` returns nothing), so this is not a mount-table leak — specifically leaked processes holding open file descriptors, plausibly including the job's own stdout/stderr pipe. This is a plausible explanation for the empty `io_log` recorded for this job on the original submission that prompted this investigation.
Reproduced twice independently on this device; both times orphans were left behind.
I have **not** proven this is the underlying cause of the original submission's `crash: Job rebooted the machine or the Checkbox agent` outcome — that could not be reproduced in isolation, possibly requiring the cumulative state of a much longer sequential test run.
## What I'm asking checkbox to own
Regardless of which upstream stress-ng behavior or platform-specific side effect ultimately causes a stressor process to hang or leak, I don't think a single stuck stressor should be able to stall an entire test plan. Two independent, unrelated hardware reports have now hit this. Suggest:
- The job (or the harness running it) enforce a hard timeout that reliably terminates the entire process group/tree spawned by `stress-ng`, not just the immediate parent, so a leaked child can't hold the job (and the session) open indefinitely.
- Once the underlying stress-ng defect(s) are identified/fixed upstream, the `stress-ng` binary bundled in the checkbox/checkbox-ce-oem snaps will need a version bump and snap rebuild.
I'm not proposing a specific implementation — just flagging that this job family currently has no defense against a stuck/leaked stressor process, and that defense would have prevented both reports below from stalling their sessions.
## Affected test case(s)
`stress/stress-ng-test-for-stressor-umount-root`
(defined in `checkbox-provider-base/units/stress/stress-ng.yaml`, template `stress-ng-test-for-stressor-{stressor}-root`)
## Environment (this report)
- CID 202605-38721, Qualcomm Technologies, Inc. Hamoa IoT EVK (X1E80100)
- Kernel `7.0.0-1013-qcom`
- `checkbox26` snap rev 164 (`stress-ng` 0.22.00, gcc 15.2.0)
- 12 CPUs online
## Related
- [LP #2166125](https://bugs.launchpad.net/baoshan/+bug/2166125) — same job, different platform (MediaTek Genio 720), different suspected mechanism, same "session gets stuck after umount stressor" symptom. Confirmed/Critical, unresolved as of filing.
- A separate upstream `stress-ng` root-cause investigation (why the `--umount` worker processes don't terminate with the parent) has not yet been filed against `ColinIanKing/stress-ng` — holding off pending further reproduction.
Contributor guide
Research direction
Start with checkbox-provider-base/units/stress/stress-ng.yaml and the harness entry point that runs the stress-ng-test-for-stressor-{stressor}-root template. Reproduce the listed command with a short timeout, then trace how the job process and its descendants are waited for. Done means a leaked or hung stressor process cannot keep the job or the remaining test plan blocked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100