test(e2e): remove remaining native-Podman false reds after #9923
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Investigation Summary
- PR #9923 merged supported native Podman activation; the final full-matrix reruns left no confirmed Podman product regression.
- The only failure unique to Podman in attempt 3 was the OpenClaw channels stop/start lane. Its Docker twin passed.
- That Podman lane failed identically in attempts 2 and 3 because the E2E directly asserted a private stopped-state cleanup helper returned `{ cleared: true }`; it instead returned `cleanup-helper-failed`.
- A second Podman-only false red appeared in attempt 2 when the cold-onboard progress assertion measured a 63-second silent gap against a 60-second guarantee. The same lane passed attempt 3 without a code change.
- Other red Podman-labeled jobs in attempt 3 also had red Docker twins and are outside this issue.
## Description
Clean up the remaining native-Podman E2E false reds observed while qualifying PR #9923.
This issue is test/harness work. The available evidence does not show that native Podman activation, channel stop/start behavior, inference, or cleanup failed as a user-visible product contract. The persistent failure comes from `test/e2e/live/channels-stop-start-helpers.ts` directly invoking `clearStoppedSandboxStateRoots` and asserting an internal result. The test reaches this assertion after the preceding channel lifecycle phases pass.
The intermittent cold-onboard failure is also a harness/progress-reporting problem: attempt 2 observed a 63-second output gap, while attempt 3 passed the same 60-second requirement without a product change.
## Reproduction Steps
1. Inspect full Docker + Podman E2E run [33594413526](https://github.com/NVIDIA/NemoClaw/actions/runs/33594413526), candidate `ebb851b756df8b74e9372f680616c634bf59316f`.
2. In attempt 2, inspect [OpenClaw channels stop/start on Podman](https://github.com/NVIDIA/NemoClaw/actions/runs/33594413526/job/100154231481).
3. In attempt 3, inspect the repeated [OpenClaw channels stop/start on Podman](https://github.com/NVIDIA/NemoClaw/actions/runs/33594413526/job/100158825576).
4. Compare the green attempt-3 [Docker twin](https://github.com/NVIDIA/NemoClaw/actions/runs/33594413526/job/100158825116).
5. For the progress flake, inspect attempt 2 [OpenClaw full E2E on Podman](https://github.com/NVIDIA/NemoClaw/actions/runs/33594413526/job/100154231401), then compare its attempt-3 success in the same workflow run.
## Environment
- GitHub Actions Ubuntu 24.04
- Native rootless Podman lane with Docker unavailable
- Managed-image workload source
- PR #9923 qualification candidate: `ebb851b756df8b74e9372f680616c634bf59316f`
- PR #9923 merged to `main` as `5c8f991d2d7273ad512946dc701b230b3b338d5a`
## Debug Output
The repeatable channels failure is:
```text
AssertionError: expected { cleared: false, …(1) } to deeply equal { cleared: true }
Expected:
{ "cleared": true }
Received:
{ "cleared": false, "failure": "cleanup-helper-failed" }
removeChannelsAndRebuild
test/e2e/live/channels-stop-start-helpers.ts:1079
```
The intermittent progress failure is:
```text
AssertionError: longest silent gap 63s exceeds the 60s guarantee:
expected 63 to be less than or equal to 60
assertColdOnboardPerformance
test/e2e/live/full-e2e.test.ts:358
```
## Expected Behavior
- The channels stop/start E2E should exercise the supported public channel-removal lifecycle rather than directly requiring a private stopped-state cleanup helper to succeed.
- The test must continue to prove channel state, provider/policy removal, sandbox stop/start, preserved unrelated state, and final cleanup on both Docker and native Podman.
- Rootless-Podman onboarding must emit observable progress within the existing 60-second guarantee, including during slow managed-image/build work.
- A test failure must retain the exact failing operation and sanitized exit/stderr evidence instead of collapsing useful diagnostics into only `cleanup-helper-failed`.
## Acceptance Criteria
- Reconcile [PR #10630](https://github.com/NVIDIA/NemoClaw/pull/10630) with the native-Podman code now on `main`; do not apply its pre-#9923 Docker assumptions blindly.
- Update `channels-stop-start` to drive the supported public channel-removal flow. Do not make the E2E call a private cleanup helper solely to manufacture stopped-state coverage.
- Preserve real behavioral assertions for OpenClaw channel removal, disabled/enabled state, provider and policy lifecycle, stop/start recovery, unrelated-state preservation, and cleanup.
- If stopped-state cleanup remains a supported provider contract after the #10630 reconciliation, cover it at the provider/integration boundary and record the sanitized helper create/start exit status and stderr. Do not keep only an opaque `cleanup-helper-failed` result.
- Fix progress emission or measurement so the Podman cold-onboard lane reliably satisfies the existing 60-second heartbeat contract. Do not merely raise the threshold, add a retry, or suppress the assertion.
- Add deterministic tests for the selected public-flow and heartbeat behavior.
- Run the exact affected Docker and native-Podman lanes. Both must pass from the same current candidate.
- Keep shared Docker/Podman inference, MCP, credential-scan, and GPU failures out of this issue.
## Scope Boundaries
- Test and E2E harness cleanup only unless new direct evidence proves a product defect.
- No change to inference routing, image publication, GPU qualification, unrelated Docker failures, or the native-Podman activation contract merged by #9923.
- No weakening of exact runtime identity, fail-closed cleanup, or credential redaction.
## Checklist
- [x] Reproduced twice for the stopped-state helper assertion.
- [x] Confirmed the Docker twin passed in attempt 3.
- [x] Confirmed the heartbeat failure passed on rerun without a code change.
- [x] Searched open and closed issues for an exact duplicate.
Contributor guide
Research direction
Reconcile PR #10630 with the current main branch, then inspect test/e2e/live/channels-stop-start-helpers.ts around removeChannelsAndRebuild and test/e2e/live/full-e2e.test.ts around assertColdOnboardPerformance. Run the linked Docker and native-Podman lanes first. Done means the public channel-removal flow, diagnostics, and 60-second heartbeat are covered by deterministic tests and both affected lanes pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, typescript
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100