canonical / canonical/checkbox

stress-ng-test-for-stressor-umount-root can stall the entire test session (leaked/hung stressor process, no enforced job-level timeout)

Open
#2,840 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
52
Forks
81
Avg merge
2d 17h
Merged PRs (30d)
69

Description

## Summary

`stress/stress-ng-test-for-stressor-umount-root` (and likely other `stress-ng-test-for-stressor-*` jobs) can leave the DUT/test session unable to continue after the stressor itself has finished, because the job has no enforced mechanism to detect and terminate a hung/leaked process tree. Two independent reports now show this stalling the remainder of a test plan on two different SoCs/kernels:

- This report: Qualcomm X1E80100 (Hamoa IoT EVK), kernel `7.0.0-1013-qcom`, server image — `stress-ng` reports the umount stressor completed successfully, but orphaned `stress-ng-umount` worker processes remain alive (reparented to init), blocked indefinitely in `ppoll()`, evidently still holding the job's output pipe open.
- Related: [LP #2166125](https://bugs.launchpad.net/baoshan/+bug/2166125) — MediaTek Genio 720, kernel `6.8.0-1006-mtk`, desktop image — same job (`stress-ng-test-for-stressor-umount-root`), a different suspected mechanism (D-Bus match-rule quota flooding from rapid mount/umount events), but the same end symptom: "Stressor umount works well, but Checkbox is stopped/killed, which causes the whole testing session to stop there."

Two different platforms, two different plausible underlying causes, same symptom family: the umount stressor job can leave the harness/session unable to proceed to the next job.

## What is proven (this report, Qualcomm platform)

Job command reproduced exactly as defined in
`checkbox-provider-base/units/stress/stress-ng.yaml` (template
`stress-ng-test-for-stressor-{stressor}-root`):

```
cd /var/tmp
stress-ng --umount 0 --timeout 30 --oom-avoid-bytes 10% --skip-silent --verbose
```
(reproduced with `--timeout 15` for faster iteration; behavior identical)

`stress-ng` itself reports full success:
```
umount: [116433] terminated (success)
metrics-check: all stressor metrics validated and sane
passed: 1: umount (1) failed: 0
successful run completed in 15.00 secs
```

But after the parent process exits, child processes remain alive, reparented to init:
```
PID PPID STAT ELAPSED COMMAND
116270 1 S 06:00 stress-ng-umoun
116271 1 S 06:00 stress-ng-umoun
116273 116270 S 06:00 stress-ng-umoun
```

Kernel stack of one such process, still blocked 6 minutes after parent exit:
```
poll_schedule_timeout.constprop.0 -> do_poll.constprop.0 -> do_sys_poll ->
__arm64_sys_ppoll -> invoke_syscall.constprop.0 -> el0_svc_common ...
```
i.e. blocked indefinitely in `ppoll()`. No leftover mount-namespace entries were found (`mount | grep stress-ng` returns nothing), so this is not a mount-table leak — specifically leaked processes holding open file descriptors, plausibly including the job's own stdout/stderr pipe. This is a plausible explanation for the empty `io_log` recorded for this job on the original submission that prompted this investigation.

Reproduced twice independently on this device; both times orphans were left behind.

I have **not** proven this is the underlying cause of the original submission's `crash: Job rebooted the machine or the Checkbox agent` outcome — that could not be reproduced in isolation, possibly requiring the cumulative state of a much longer sequential test run.

## What I'm asking checkbox to own

Regardless of which upstream stress-ng behavior or platform-specific side effect ultimately causes a stressor process to hang or leak, I don't think a single stuck stressor should be able to stall an entire test plan. Two independent, unrelated hardware reports have now hit this. Suggest:

- The job (or the harness running it) enforce a hard timeout that reliably terminates the entire process group/tree spawned by `stress-ng`, not just the immediate parent, so a leaked child can't hold the job (and the session) open indefinitely.
- Once the underlying stress-ng defect(s) are identified/fixed upstream, the `stress-ng` binary bundled in the checkbox/checkbox-ce-oem snaps will need a version bump and snap rebuild.

I'm not proposing a specific implementation — just flagging that this job family currently has no defense against a stuck/leaked stressor process, and that defense would have prevented both reports below from stalling their sessions.

## Affected test case(s)

`stress/stress-ng-test-for-stressor-umount-root`
(defined in `checkbox-provider-base/units/stress/stress-ng.yaml`, template `stress-ng-test-for-stressor-{stressor}-root`)

## Environment (this report)

- CID 202605-38721, Qualcomm Technologies, Inc. Hamoa IoT EVK (X1E80100)
- Kernel `7.0.0-1013-qcom`
- `checkbox26` snap rev 164 (`stress-ng` 0.22.00, gcc 15.2.0)
- 12 CPUs online

## Related

- [LP #2166125](https://bugs.launchpad.net/baoshan/+bug/2166125) — same job, different platform (MediaTek Genio 720), different suspected mechanism, same "session gets stuck after umount stressor" symptom. Confirmed/Critical, unresolved as of filing.
- A separate upstream `stress-ng` root-cause investigation (why the `--umount` worker processes don't terminate with the parent) has not yet been filed against `ColinIanKing/stress-ng` — holding off pending further reproduction.

Contributor guide

Open the contributing guide

Research direction

Start with checkbox-provider-base/units/stress/stress-ng.yaml and the harness entry point that runs the stress-ng-test-for-stressor-{stressor}-root template. Reproduce the listed command with a short timeout, then trace how the job process and its descendants are waited for. Done means a leaked or hung stressor process cannot keep the job or the remaining test plan blocked.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.