persistent volume: .nfs* orphan files leak storage when sandbox VMs are forcibly killed

Problem

When a sandbox VM is forcibly killed or crashes while holding open file descriptors on files inside a persistent volume, the NFS silly-rename mechanism leaves .nfs* orphan files on the NFS server that are never cleaned up.

Observed:

/mnt/images$ ls -ltra
-rw-r--r-- 1 root root 5368709120 Jul 29 17:13 .nfs000000019000ee9800000001
-rw-r--r-- 1 root root 5368709120 Aug  4 18:12 data_dadd413a-7f01-4a0a-affb-2f8aba247051.img

A 5 GiB workspace image from Jul 29 remained as .nfs000000019000ee9800000001 and was never removed — wasting the full 5 GiB of persistent volume storage indefinitely.

Root Cause

The NFS silly-rename mechanism works like this:

1. A sandbox VM (NFS client) holds an open fd on a file inside the persistent volume (e.g. data_<sandbox-id>.img)
2. Orchestrator cleanup calls os.Remove() on the file (via chroot/fs.go:Remove)
3. The NFS server receives a REMOVE RPC — but the NFS client kernel inside the VM detects the file is still open locally
4. The NFS client replaces the REMOVE with a RENAME → .nfsXXXXXX, keeping the data accessible until the fd closes
5. The VM is killed/crashes before it can send the final REMOVE for the .nfs* file
6. The .nfs* file remains on the NFS server permanently — no process will ever close that fd or send the REMOVE

The cleanup in OnNetworkRelease (packages/orchestrator/pkg/nfsproxy/chroot/nfs.go:95) closes the orchestrator-side chroot but does not scan for or remove leftover .nfs* files in the volume directory:

func (h *NFSHandler) OnNetworkRelease(ctx context.Context, sbx *sandbox.Sandbox) {
    // closes the chroot — but .nfs* orphans in the volume dir are not removed
    for _, chroot := range chroots {
        err := chroot.Close()
        ...
    }
}

The existing reclaim utilities (ReclaimSandboxFiles, clean-nfs-cache job) target different paths (/tmp, sandbox cache dir, NFS chunk cache) and do not cover persistent volume directories.

Impact

• Storage leak: each incident leaves behind a file as large as the original (workspace disk images are typically several GiB)
• Cumulative: every non-graceful VM termination on a sandbox using a persistent volume contributes one orphan file
• No automatic recovery: without a cleanup mechanism targeting .nfs* files in volume dirs, the orphans accumulate until manual intervention

Suggested Fix

After OnNetworkRelease (or as a periodic cleanup job), scan the released volume directory for .nfs* files and remove them:

func (h *NFSHandler) OnNetworkRelease(ctx context.Context, sbx *sandbox.Sandbox) {
    // existing chroot close logic ...

    // clean up NFS silly-rename orphans left by killed VMs
    for _, mount := range sbx.Config.VolumeMounts {
        volPath, err := h.builder.BuildVolumePath(mount.Type, teamID, mount.ID)
        if err != nil {
            continue
        }
        cleanNFSOrphans(ctx, volPath)
    }
}

Where cleanNFSOrphans removes files matching .nfs* in the directory. Alternatively, extend the existing clean-nfs-cache periodic job to also sweep persistent volume directories.

Code Paths

Start by reading packages/orchestrator/pkg/nfsproxy/chroot/nfs.go at OnNetworkRelease and packages/orchestrator/pkg/chrooted/builder.go at BuildVolumePath; compare the existing reclaim logic in packages/shared/pkg/storage/sandbox.go. Done means orphan .nfs* files are safely removed from released persistent volume directories without changing cleanup of unrelated paths.