NVIDIA / NVIDIA/NemoClaw

test(e2e): remove remaining native-Podman false reds after #9923

Open
#10,865 0 comments 0 reactions 0 assignees View on GitHub
area: e2e integration: openclaw platform: container
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Investigation Summary

- PR #9923 merged supported native Podman activation; the final full-matrix reruns left no confirmed Podman product regression.
- The only failure unique to Podman in attempt 3 was the OpenClaw channels stop/start lane. Its Docker twin passed.
- That Podman lane failed identically in attempts 2 and 3 because the E2E directly asserted a private stopped-state cleanup helper returned `{ cleared: true }`; it instead returned `cleanup-helper-failed`.
- A second Podman-only false red appeared in attempt 2 when the cold-onboard progress assertion measured a 63-second silent gap against a 60-second guarantee. The same lane passed attempt 3 without a code change.
- Other red Podman-labeled jobs in attempt 3 also had red Docker twins and are outside this issue.

## Description

Clean up the remaining native-Podman E2E false reds observed while qualifying PR #9923.

This issue is test/harness work. The available evidence does not show that native Podman activation, channel stop/start behavior, inference, or cleanup failed as a user-visible product contract. The persistent failure comes from `test/e2e/live/channels-stop-start-helpers.ts` directly invoking `clearStoppedSandboxStateRoots` and asserting an internal result. The test reaches this assertion after the preceding channel lifecycle phases pass.

The intermittent cold-onboard failure is also a harness/progress-reporting problem: attempt 2 observed a 63-second output gap, while attempt 3 passed the same 60-second requirement without a product change.

## Reproduction Steps

1. Inspect full Docker + Podman E2E run [33594413526](https://github.com/NVIDIA/NemoClaw/actions/runs/33594413526), candidate `ebb851b756df8b74e9372f680616c634bf59316f`.
2. In attempt 2, inspect [OpenClaw channels stop/start on Podman](https://github.com/NVIDIA/NemoClaw/actions/runs/33594413526/job/100154231481).
3. In attempt 3, inspect the repeated [OpenClaw channels stop/start on Podman](https://github.com/NVIDIA/NemoClaw/actions/runs/33594413526/job/100158825576).
4. Compare the green attempt-3 [Docker twin](https://github.com/NVIDIA/NemoClaw/actions/runs/33594413526/job/100158825116).
5. For the progress flake, inspect attempt 2 [OpenClaw full E2E on Podman](https://github.com/NVIDIA/NemoClaw/actions/runs/33594413526/job/100154231401), then compare its attempt-3 success in the same workflow run.

## Environment

- GitHub Actions Ubuntu 24.04
- Native rootless Podman lane with Docker unavailable
- Managed-image workload source
- PR #9923 qualification candidate: `ebb851b756df8b74e9372f680616c634bf59316f`
- PR #9923 merged to `main` as `5c8f991d2d7273ad512946dc701b230b3b338d5a`

## Debug Output

The repeatable channels failure is:

```text
AssertionError: expected { cleared: false, …(1) } to deeply equal { cleared: true }

Expected:
{ "cleared": true }

Received:
{ "cleared": false, "failure": "cleanup-helper-failed" }

removeChannelsAndRebuild
test/e2e/live/channels-stop-start-helpers.ts:1079
```

The intermittent progress failure is:

```text
AssertionError: longest silent gap 63s exceeds the 60s guarantee:
expected 63 to be less than or equal to 60

assertColdOnboardPerformance
test/e2e/live/full-e2e.test.ts:358
```

## Expected Behavior

- The channels stop/start E2E should exercise the supported public channel-removal lifecycle rather than directly requiring a private stopped-state cleanup helper to succeed.
- The test must continue to prove channel state, provider/policy removal, sandbox stop/start, preserved unrelated state, and final cleanup on both Docker and native Podman.
- Rootless-Podman onboarding must emit observable progress within the existing 60-second guarantee, including during slow managed-image/build work.
- A test failure must retain the exact failing operation and sanitized exit/stderr evidence instead of collapsing useful diagnostics into only `cleanup-helper-failed`.

## Acceptance Criteria

- Reconcile [PR #10630](https://github.com/NVIDIA/NemoClaw/pull/10630) with the native-Podman code now on `main`; do not apply its pre-#9923 Docker assumptions blindly.
- Update `channels-stop-start` to drive the supported public channel-removal flow. Do not make the E2E call a private cleanup helper solely to manufacture stopped-state coverage.
- Preserve real behavioral assertions for OpenClaw channel removal, disabled/enabled state, provider and policy lifecycle, stop/start recovery, unrelated-state preservation, and cleanup.
- If stopped-state cleanup remains a supported provider contract after the #10630 reconciliation, cover it at the provider/integration boundary and record the sanitized helper create/start exit status and stderr. Do not keep only an opaque `cleanup-helper-failed` result.
- Fix progress emission or measurement so the Podman cold-onboard lane reliably satisfies the existing 60-second heartbeat contract. Do not merely raise the threshold, add a retry, or suppress the assertion.
- Add deterministic tests for the selected public-flow and heartbeat behavior.
- Run the exact affected Docker and native-Podman lanes. Both must pass from the same current candidate.
- Keep shared Docker/Podman inference, MCP, credential-scan, and GPU failures out of this issue.

## Scope Boundaries

- Test and E2E harness cleanup only unless new direct evidence proves a product defect.
- No change to inference routing, image publication, GPU qualification, unrelated Docker failures, or the native-Podman activation contract merged by #9923.
- No weakening of exact runtime identity, fail-closed cleanup, or credential redaction.

## Checklist

- [x] Reproduced twice for the stopped-state helper assertion.
- [x] Confirmed the Docker twin passed in attempt 3.
- [x] Confirmed the heartbeat failure passed on rerun without a code change.
- [x] Searched open and closed issues for an exact duplicate.

Contributor guide

Open the contributing guide

Research direction

Reconcile PR #10630 with the current main branch, then inspect test/e2e/live/channels-stop-start-helpers.ts around removeChannelsAndRebuild and test/e2e/live/full-e2e.test.ts around assertColdOnboardPerformance. Run the linked Docker and native-Podman lanes first. Done means the public channel-removal flow, diagnostics, and 60-second heartbeat are covered by deterministic tests and both affected lanes pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.