NVIDIA / NVIDIA/NemoClaw

e2e(gateway): guard-chain recovery intermittently health-times out after recreation

Open
#10,977 0 comments 0 reactions 1 assignee Claimed by @cjagwani View on GitHub
area: ci area: e2e area: observability area: sandbox ci-failure
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Investigation Summary

- [Gateway guard-chain recovery](https://github.com/NVIDIA/NemoClaw/actions/runs/33730208977/job/100568886794) fails on `3076188279c0ea657ec77aeeabebf1f82061eaaf` with `GATEWAY_HEALTH_TIMEOUT` after legacy recovery/recreation.
- Failure evidence includes invalid/stale PID or process-stage observations before the health deadline.
- The [same target passes](https://github.com/NVIDIA/NemoClaw/actions/runs/33761515943/job/100678908962) on later main candidate `eb3fa2e5097389d7dd8e2643552788ff16d8f14a`.
- Closed issue #7484 documents the broader supervised-gateway timeout family; this tracker covers recurrence in the guard-chain recreation E2E.

## Description

Gateway guard recovery is timing-sensitive after recreation. The outer controller can time out without identifying whether the replacement process, PID handoff, listener, or health surface lagged.

Expected: the replacement gateway reaches a stable health state, with actionable stage-specific evidence if it does not.

Actual: the E2E receives a generic health timeout and passes on rerun.

## Reproduction Steps

1. Run the guard-chain recreation Docker E2E repeatedly under CI load.
2. Allow the target to recreate and recover the legacy gateway.
3. Observe intermittent PID/process transition failure followed by `GATEWAY_HEALTH_TIMEOUT`.

## Environment

- GitHub Actions Docker E2E
- Failed run: [33730208977](https://github.com/NVIDIA/NemoClaw/actions/runs/33730208977)
- Passing comparison: [33761515943](https://github.com/NVIDIA/NemoClaw/actions/runs/33761515943)

## Acceptance Criteria

- Record supervisor PID, child PID, listener readiness, and health stage at failure.
- Make replacement PID handoff robust against expected startup latency.
- Demonstrate repeated guard-chain recovery passes under CI load.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.