agent-substrate / agent-substrate/substrate
[P3] Compress temp files in os.TempDir() not cleaned on atelet crash — /tmp fills up
- Dominant language
- Go
- Stars
- 1.8k
- Forks
- 316
- Avg merge
- 2d 43m
- Merged PRs (30d)
- 287
Description
**Severity:** P3 (slow disk leak in /tmp; accelerates ENOSPC via ISSUE-007)
**Component:** Node Layer — `cmd/atelet/internal/ategcs/objects.go`
**Audit ID:** NL-9
---
## Summary
`sendBufferedZstd` (used for S3/non-streaming GCS uploads) creates a temp file via
`os.CreateTemp("", "substrate-upload-compress-")` in `os.TempDir()` (typically `/tmp`).
The deferred `os.Remove` cleanup only runs if the process exits normally. A SIGKILL
(OOM kill, node restart, crash) leaves potentially multi-GiB temp files in `/tmp`.
There is no startup sweep to clean these up. Repeated crashes during large checkpoint
uploads can exhaust `/tmp`.
---
## Root Cause
**File:** `cmd/atelet/internal/ategcs/objects.go` lines 186–212
```go
func sendBufferedZstd(ctx context.Context, ...) error {
tmpFile, err := os.CreateTemp("", "substrate-upload-compress-")
if err != nil { return err }
defer os.Remove(tmpFile.Name()) // ← NOT called on SIGKILL
// Write compressed data to tmpFile (may be hundreds of MB)
// Upload tmpFile to S3/GCS
return nil
}
```
No code in `atelet/main.go`'s startup path sweeps for leftover `substrate-upload-compress-*`
files.
---
## Steps to Reproduce
```bash
# Start a large checkpoint upload (use a big actor snapshot)
kubectl ate suspend actor large-actor -a demo &
# Kill atelet mid-upload
kubectl exec -n ate-system -- kill -9 1
# Check /tmp
kubectl exec -n ate-system -- ls -lh /tmp/substrate-upload-compress-*
# Files remain, potentially several GB each
```
---
## Suggested Fix
On atelet startup, add a sweep:
```go
func sweepCompressTempFiles() {
pattern := filepath.Join(os.TempDir(), "substrate-upload-compress-*")
files, _ := filepath.Glob(pattern)
for _, f := range files {
_ = os.Remove(f)
}
}
```
Call `sweepCompressTempFiles()` in `main.go` before the gRPC server starts.
Alternatively, create temp files inside `BasePath` (the atelet's dedicated volume)
rather than `os.TempDir()`, so they are isolated from system `/tmp` and can be swept
by the existing per-actor cleanup paths.
Contributor guide
Assessment
This issue has not been assessed yet.