stacklok / stacklok/mecatl

Bash disappears from live sessions after managed temporary namespace deletion

Open
#1,337 0 comments 0 reactions 1 assignee View on GitHub

@JAORMX is already working on this.

Since Sep 10, 2026.

bug
Dominant language
Go
Stars
152
Forks
16
Avg merge
14h 48m
Merged PRs (30d)
536

Description

So, Bash works, agents use it successfully, and then the same live session suddenly says that no shell exists. We now have a confirmed reproduction. The managed temporary namespace was deleted while mecatui remained alive, and every later run in that process silently lost Bash.

Impact

Severity: high availability / medium integrity risk.

The affected process keeps running, but its tool surface changes underneath existing sessions:

  • Main-session Bash becomes unavailable.
  • Subagents can still receive Edit, Write, Read, and MCP tools while Bash is removed.
  • Agents may continue partially, skip validation/commit/push steps, and finish cleanly despite lacking a tool required by the task.
  • Persisted authority can still contain Bash even though the active request has removed it, making snapshots alone misleading during diagnosis.

No privilege escalation or direct data loss has been confirmed.

Confirmed incident

The clearest affected session was:

session: 36cf3e5f3b1869893087e769aba8e645
child:   subagent-36cf3e5f3b1869893087e769aba8e645-call_Ig8uBLsyjzVrUSiEo3fdxFEH
server:  PID 1890478

The same mecatui process remained alive throughout the incident. It did not restart or change session placement.

Before the failure:

  • Request manifests advertised Bash.
  • Children made 134 Bash calls, 125 successfully.
  • The final confirmed Bash-capable child completed at 09:28:35 +03.

At 12:25:42 +03, Fish history records the external trigger:

rm -rf /tmp/mecatl

The live server retained an open descriptor to the removed namespace:

/tmp/mecatl (deleted)

A new /tmp/mecatl pathname was created later, but the old process remained attached to the unlinked inode.

At 13:08:19 +03, the next placement/runner construction first reported:

managedtemp: workspaces: statat workspaces: no such file or directory

Every later request manifest in that session classified Bash as:

mount_unavailable:catalog

The affected child made Read/Grep/ListDir calls but zero Bash calls. A later turn in the same session asking whether Bash was available produced the same managed-temp failure and the same mount_unavailable decision.

This is a real loss between separate prompt runs of one session, not a model hallucination and not an embedded-server restart.

Regression boundary

The failure surface was introduced by PR #994, commit be2663dfc (feat(temp): add managed temporary command leases), first shipped in v0.0.26 on September 8. That matches the first observed incidents this week.

PR #996's exact per-run session placement reattachment exposes the failure on the next prompt, but it is not the root cause. Reattachment is correct. Removing it would only hide the stale namespace until the next managed lease allocation failed.

Root cause

PR #994 made command-runner construction depend on a process-shared managed namespace, normally:

/tmp/mecatl/workspaces/

managedtemp.Namespace opens the root once and retains an os.Root handle. Later runner construction calls OpenWorkspace, which validates and opens workspaces through that retained handle.

After the pathname is removed:

  1. The live process still holds the old, now-unlinked namespace handle.
  2. Recreating /tmp/mecatl creates a different inode that the old handle cannot see.
  3. OpenWorkspace permanently fails while resolving workspaces through the stale handle.
  4. Command-runner construction logs a warning and returns nil.
  5. Placement reattachment still succeeds with a shell-less environment.
  6. Request assembly removes Bash as mount_unavailable.

Relevant code:

  • internal/adapter/managedtemp/namespace_unix.go (Open, OpenWorkspace)
  • internal/app/temporary_storage.go (managedTemporaryStorage.workspace)
  • internal/app/build.go (newCommandRunnerForRoot, runner/catalog construction)
  • internal/app/placement.go (bindLocal, per-run reattachment)
  • engine/agent/loop.go (live-environment tool filtering)
  • engine/agent/request_manifest.go (mount_unavailable evidence)

On current main, the key ranges are approximately:

  • internal/adapter/managedtemp/namespace_unix.go:42-120
  • internal/app/build.go:6140-6182
  • internal/app/placement.go:53-70
  • internal/app/placement.go:175-195
  • engine/agent/loop.go:2573-2686
  • engine/agent/request_manifest.go:46-90

The managed-temp reaper is not the deletion actor. Its deletion authority stops at validated cmd-* and job-* allocation directories. Fedora tmpfiles is also not responsible: /tmp has a 10-day policy, the directory was only a few days old, and the timer did not run at the incident time.

The external rm is the trigger, but losing disposable temporary storage must not permanently corrupt a live harness. The implementation problems are:

  • The managed namespace cannot safely reopen or recreate itself after pathname loss/replacement.
  • A runner-construction failure silently turns a previously Bash-enabled local placement into a valid shell-less placement.
  • Existing sessions and direct-write children can continue with a reduced tool surface and reach a clean terminal state.

Reproduction

Use an isolated managed root. Do not run this against a shared development process.

  1. Start mecated or embedded mecatui with Bash and managed temporary storage enabled.

  2. Create a normal filesystem-backed session.

  3. Run Bash successfully.

  4. Keep the server process alive.

  5. From another shell, remove the configured managed root:

    rm -rf /path/to/isolated/mecatl-managed-root
    
  6. Continue the same session with another prompt.

  7. Inspect diagnostics and the request manifest.

Actual result:

  • OpenWorkspace fails with statat workspaces: no such file or directory.
  • Runner construction returns nil.
  • Bash is classified mount_unavailable and omitted from the request.
  • The session continues as though shell-less operation were intentional.

Expected result:

  • Mecatl safely reopens/recreates the exact configured private namespace and retains Bash, or
  • reattachment fails explicitly with an actionable managed-temporary-storage error.

It must not silently change an existing session's configured capability surface.

Acceptance criteria

  • Reproduce namespace deletion after at least one managed workspace handle is already open.
  • Safely reopen or recreate the exact configured managed root and required workspaces structure.
  • Retire stale handles after recovery so the process no longer uses an unlinked namespace.
  • Preserve owner, mode, no-link, replacement, and confinement checks from PR #994.
  • Reject unsafe replacements such as a symlink, wrong owner, wrong mode, or wrong type without mutating them.
  • If recovery is impossible, fail local placement reattachment explicitly. Do not return a valid environment with a nil runner when Bash is configured.
  • Preserve intentional shell-less behavior for no-FS sessions, explicit no-Bash configuration, ACP, and other shell-less placements.
  • Prove a continued session advertises and executes Bash after successful recovery.
  • Cover main sessions and direct-write Subagents after recovery.
  • Cover concurrent reattachment, allocation, and sweep during namespace loss.
  • Verify request manifests do not silently transition Bash from advertised to mount_unavailable after recoverable namespace loss.
  • Add an integration test proving a shell-required direct-write child either receives Bash or fails before starting. It must not return a clean, incomplete result.

Additional diagnostic distinction

We have also seen agents claim they lack shell when Bash remained advertised and they never attempted it. That is a separate model/tool-use failure.

For this bug, the request manifest is decisive:

  • persisted authority containing Bash is only an authorization ceiling;
  • mount_unavailable:catalog in the active request proves real environment-level capability loss;
  • Bash being advertised with a non-nil runner, followed only by model prose claiming otherwise, is not this failure.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.