NVIDIA / NVIDIA/NemoClaw

Hermes GPU fallback refuses an exact managed cleanup handoff

Open
#9,935 2 comments 0 reactions 0 assignees View on GitHub
area: onboarding area: sandbox integration: hermes platform: container security v0.0.127
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Investigation summary

PR #9323's exact unfiltered E2E run at commit `5edd54a44cdc124b5ecf06ff3cda3e134803e857` fails only the Hermes native-to-compatibility fallback scenario. Native GPU startup and compatibility-only startup both pass.

The fallback failure is distinct from #9486. The #9512 repair is present, no shared-state metadata mismatch or duplicate managed container remains, and strict cleanup destroys the sandbox and releases the gateway port.

## Root cause

Commit `d33393958` added `cleanupNativeGpuFailureForFallback()`. When the managed lifecycle returns an exact owner-bound cleanup handoff, the helper returns `safe: false`. The caller then refuses the compatibility retry even though the exact managed cleanup path completed safely.

The prior candidate `c073313` contains #9512 and passes the same fallback scenario. The regression begins with the later #9323 cleanup change.

## Evidence

- Failing exact run: https://github.com/NVIDIA/NemoClaw/actions/runs/32512936706/job/96868526146
- Passing native scenario at the same commit: https://github.com/NVIDIA/NemoClaw/actions/runs/32512936706/job/96868526030
- Passing compatibility-only scenario at the same commit: https://github.com/NVIDIA/NemoClaw/actions/runs/32512936706/job/96868526154
- Passing fallback scenario before the regression: https://github.com/NVIDIA/NemoClaw/actions/runs/32412828260/job/96584043003

## Accepted scope

- Preserve exact managed lifecycle ownership and fail-closed cleanup.
- Treat an exact owner-bound cleanup handoff that completes and reconciles external state as safe for the one intended compatibility retry.
- Do not fall back to mutable sandbox-name container selection.
- Do not add a broad retry or weaken strict cleanup.
- Add deterministic coverage for the exact cleanup-handoff result and retry decision.

## Acceptance evidence

- Focused lifecycle and Hermes GPU fallback tests pass.
- A fresh exact-head `hermes-gpu-startup` fallback E2E passes on #9323 after the repair.
- Cleanup still destroys only transaction-owned resources and releases the gateway port.

Prerequisite for #9140 and PR #9323.

Contributor guide

Open the contributing guide

Research direction

Start with the focused lifecycle and Hermes GPU fallback tests, then inspect the hermes-gpu-startup fallback E2E and the cleanup handoff result used by the retry decision. Done means deterministic coverage passes, the exact-head fallback E2E passes, and cleanup still destroys only transaction-owned resources while releasing the gateway port.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.