agent-substrate / agent-substrate/substrate

[P3] Compress temp files in os.TempDir() not cleaned on atelet crash — /tmp fills up

未關閉
#615 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/node kind/bug prio/P2
主要語言
Go
星號
1.8k
分支
316
平均合併
2 天 43 分鐘
30 天內合併 PR
287

描述

**Severity:** P3 (slow disk leak in /tmp; accelerates ENOSPC via ISSUE-007)
**Component:** Node Layer — `cmd/atelet/internal/ategcs/objects.go`
**Audit ID:** NL-9

---

## Summary

`sendBufferedZstd` (used for S3/non-streaming GCS uploads) creates a temp file via
`os.CreateTemp("", "substrate-upload-compress-")` in `os.TempDir()` (typically `/tmp`).
The deferred `os.Remove` cleanup only runs if the process exits normally. A SIGKILL
(OOM kill, node restart, crash) leaves potentially multi-GiB temp files in `/tmp`.
There is no startup sweep to clean these up. Repeated crashes during large checkpoint
uploads can exhaust `/tmp`.

---

## Root Cause

**File:** `cmd/atelet/internal/ategcs/objects.go` lines 186–212

```go
func sendBufferedZstd(ctx context.Context, ...) error {
tmpFile, err := os.CreateTemp("", "substrate-upload-compress-")
if err != nil { return err }
defer os.Remove(tmpFile.Name()) // ← NOT called on SIGKILL

// Write compressed data to tmpFile (may be hundreds of MB)
// Upload tmpFile to S3/GCS
return nil
}
```

No code in `atelet/main.go`'s startup path sweeps for leftover `substrate-upload-compress-*`
files.

---

## Steps to Reproduce

```bash
# Start a large checkpoint upload (use a big actor snapshot)
kubectl ate suspend actor large-actor -a demo &

# Kill atelet mid-upload
kubectl exec -n ate-system -- kill -9 1

# Check /tmp
kubectl exec -n ate-system -- ls -lh /tmp/substrate-upload-compress-*
# Files remain, potentially several GB each
```

---

## Suggested Fix

On atelet startup, add a sweep:

```go
func sweepCompressTempFiles() {
pattern := filepath.Join(os.TempDir(), "substrate-upload-compress-*")
files, _ := filepath.Glob(pattern)
for _, f := range files {
_ = os.Remove(f)
}
}
```

Call `sweepCompressTempFiles()` in `main.go` before the gRPC server starts.

Alternatively, create temp files inside `BasePath` (the atelet's dedicated volume)
rather than `os.TempDir()`, so they are isolated from system `/tmp` and can be swept
by the existing per-actor cleanup paths.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。