agent-substrate / agent-substrate/substrate

[P3] Compress temp files in os.TempDir() not cleaned on atelet crash — /tmp fills up

Open
#615 1 comment 0 reactions 0 assignees View on GitHub
area/node kind/bug prio/P2
Dominant language
Go
Stars
1.8k
Forks
316
Avg merge
2d 43m
Merged PRs (30d)
287

Description

**Severity:** P3 (slow disk leak in /tmp; accelerates ENOSPC via ISSUE-007)
**Component:** Node Layer — `cmd/atelet/internal/ategcs/objects.go`
**Audit ID:** NL-9

---

## Summary

`sendBufferedZstd` (used for S3/non-streaming GCS uploads) creates a temp file via
`os.CreateTemp("", "substrate-upload-compress-")` in `os.TempDir()` (typically `/tmp`).
The deferred `os.Remove` cleanup only runs if the process exits normally. A SIGKILL
(OOM kill, node restart, crash) leaves potentially multi-GiB temp files in `/tmp`.
There is no startup sweep to clean these up. Repeated crashes during large checkpoint
uploads can exhaust `/tmp`.

---

## Root Cause

**File:** `cmd/atelet/internal/ategcs/objects.go` lines 186–212

```go
func sendBufferedZstd(ctx context.Context, ...) error {
tmpFile, err := os.CreateTemp("", "substrate-upload-compress-")
if err != nil { return err }
defer os.Remove(tmpFile.Name()) // ← NOT called on SIGKILL

// Write compressed data to tmpFile (may be hundreds of MB)
// Upload tmpFile to S3/GCS
return nil
}
```

No code in `atelet/main.go`'s startup path sweeps for leftover `substrate-upload-compress-*`
files.

---

## Steps to Reproduce

```bash
# Start a large checkpoint upload (use a big actor snapshot)
kubectl ate suspend actor large-actor -a demo &

# Kill atelet mid-upload
kubectl exec -n ate-system -- kill -9 1

# Check /tmp
kubectl exec -n ate-system -- ls -lh /tmp/substrate-upload-compress-*
# Files remain, potentially several GB each
```

---

## Suggested Fix

On atelet startup, add a sweep:

```go
func sweepCompressTempFiles() {
pattern := filepath.Join(os.TempDir(), "substrate-upload-compress-*")
files, _ := filepath.Glob(pattern)
for _, f := range files {
_ = os.Remove(f)
}
}
```

Call `sweepCompressTempFiles()` in `main.go` before the gRPC server starts.

Alternatively, create temp files inside `BasePath` (the atelet's dedicated volume)
rather than `os.TempDir()`, so they are isolated from system `/tmp` and can be swept
by the existing per-actor cleanup paths.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.