[Tunnel] cloudflared tunnel is stopped by unrelated sandbox destroy/recreate, and `tunnel status` detection is keyed to the sandbox pid directory (false "stopped" / invisible orphan)
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
### Investigation Summary
- A running named tunnel (cloudflared via nemoclaw-tunnel.service, token in
~/.nemoclaw/tunnel.env) was killed by `nemoclaw destroy --force` /
recreate runs, including for sandboxes with no apparent relation to the
tunnel's public hostname; the public URL then failed with Cloudflare 530
(error 1033) until the tunnel service was restarted manually.
- `nemoclaw tunnel status` reports "stopped / no cloudflared process" while a
cloudflared process is demonstrably running (pgrep shows it), and conversely
showed a running PID only when querying through the correct pid directory —
the pid file lives at /tmp/nemoclaw-services-/cloudflared.pid, keyed
to the sandbox that was default when the service started.
- A leftover `cloudflared tunnel run` from an earlier manual attempt (6h+ old)
was invisible to the CLI (status said "stopped"); it was carrying the public
hostname the whole time, and had to be found via pgrep and killed manually
before the service-managed tunnel could take over cleanly.
- Net effect: every sandbox destroy/recreate silently takes the public
dashboard URL down, with no message tying the two together
(destroy printed only "[services] cloudflared was not running" — a false
negative when the tunnel actually was running).
### Description
The cloudflared tunnel's lifecycle is entangled with individual sandbox
lifecycle cleanup. Because the service pid/log files are keyed to a sandbox
directory (/tmp/nemoclaw-services-/), destroy/recreate operations
either (a) stop the tunnel when they clean up the owning sandbox's directory,
or (b) report "cloudflared was not running" and leave a stale process running
invisibly to the CLI. Since the tunnel's public hostnames (configured in the
Cloudflare dashboard) can point at ANY sandbox's port, stopping it during an
unrelated sandbox's cleanup is surprising and takes public URLs down silently.
Expected: tunnel lifecycle is independent of per-sandbox destroy/recreate
(the tunnel routes to origins defined in the Cloudflare dashboard, not owned
by one sandbox), or at least: cleanup of one sandbox never stops a tunnel
serving other hostnames, `tunnel status` reflects the real process state
regardless of which pid directory is consulted, and a pre-existing
cloudflared process is detected and reported instead of shadowed.
Workaround (verified): after any sandbox destroy/recreate,
systemctl --user restart nemoclaw-tunnel.service
and verify with `nemoclaw tunnel status` + a curl of the public hostname.
### Reproduction Steps
1. Set up a named tunnel: token in ~/.nemoclaw/tunnel.env,
`systemctl --user start nemoclaw-tunnel.service` (or `nemoclaw tunnel
start`); add a public hostname in the Cloudflare dashboard pointing at a
sandbox dashboard port; confirm the URL serves.
2. Run `nemoclaw destroy --force` (or a recreate) for a sandbox.
3. Observe either: the running cloudflared process is gone and the public URL
now returns 530/error 1033 (tunnel killed), or destroy prints
"[services] cloudflared was not running" although `pgrep -a cloudflared`
shows a live process (false negative / invisible process).
4. `nemoclaw tunnel status` does not agree with the actual process state in
either case until the service is restarted through its unit.
### Environment
- OS: Fedora Linux 44 (Server Edition), kernel 7.1.13-200.fc44.x86_64, x86_64
- GPU: NVIDIA GeForce RTX 4090 (driver 610.57.04), Docker CDI GPU passthrough
- Docker: Docker Engine 29.7.2 (docker driver)
- NemoClaw / NemoHermes: v0.0.109 -> v0.0.123 (upgraded 2026-09-12)
- OpenShell: 0.0.101 -> 0.0.106 (upgraded by the installer during upgrade-sandboxes)
- Hermes agent runtime: 0.19.0 -> 0.20.6 (sandbox image rebuild)
- Node.js: v24.18.0
- Layout: 2 sandboxes (leryl-01, frankiebox-02) sharing one gateway "nemoclaw" on default port 8080 and one host-local llama.cpp server (llama-cpp-local, ornith:9b,
- 127.0.0.1:8081); each sandbox has a telegram channel.
### Debug Output
nemoclaw-debug.tar.gz attached where available. The sequence below is captured
verbatim from this host's session (2026-09-12).
[nemoclaw-debug-postfix.tar.gz](https://github.com/user-attachments/files/32147269/nemoclaw-debug-postfix.tar.gz)
[nemoclaw-debug.tar.gz](https://github.com/user-attachments/files/32147268/nemoclaw-debug.tar.gz)
### Logs
```shell
1. Stale orphan invisible to the CLI (process alive since the previous day):
$ pgrep -a cloudflared
1541294 cloudflared tunnel run
$ nemoclaw tunnel status
● cloudflared (stopped)
no cloudflared process; run `nemoclaw tunnel start` to start it
2. False negative during an unrelated sandbox's destroy (tunnel was running):
$ nemoclaw leryl-01 destroy --force
[services] cloudflared was not running
(immediately after: pgrep -a cloudflared -> 1930483 cloudflared tunnel run,
tunnel still serving)
3. Tunnel actually killed by the default sandbox's destroy/recreate; public
URLs dead afterwards:
$ curl -sS -o /dev/null -w "%{http_code}" https://hermes.michaelleroy.me/
530 # Cloudflare error 1033 (tunnel down)
$ pgrep -a cloudflared
(no output)
$ systemctl --user restart nemoclaw-tunnel.service
$ curl -sS -o /dev/null -w "%{http_code}" https://hermes.michaelleroy.me/
200
The same kill-then-restart cycle was required after each of the two
recreate passes during this session.
```
### Checklist
- [x] I confirmed this bug is reproducible
- [x] I searched existing issues and this is not a duplicate
Contributor guide
Research direction
Start at the `nemoclaw tunnel status` and sandbox destroy/recreate entry points, then trace how `/tmp/nemoclaw-services-/cloudflared.pid` and `nemoclaw-tunnel.service` are used during cleanup and status checks. Reproduce with `pgrep -a cloudflared`, `nemoclaw tunnel status`, and a sandbox destroy; done means unrelated cleanup does not stop the tunnel and status reports an existing process correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cli, infrastructure, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100