hashicorp / hashicorp/raft

FileSnapshotSink: failed write leaves state.bin open, and Cancel() leaves the .tmp directory on disk

Open Beginner friendly
#714 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
9.1k
Forks
1.1k
Avg merge
3h 27m
Merged PRs (30d)
2

Description

If writing a snapshot fails (e.g. the disk fills up), `FileSnapshotSink` doesn't clean up properly:

- `finalize()` returns early when `Flush()` or `Sync()` fails, so `state.bin` never gets closed. This affects both `Close()` and `Cancel()` ([file_snapshot.go#L470-L478](https://github.com/hashicorp/raft/blob/v1.8.0/file_snapshot.go#L470-L478)).
- `Cancel()` returns that error before `os.RemoveAll(s.dir)`, so the partial `.tmp` directory stays on disk ([file_snapshot.go#L458-L464](https://github.com/hashicorp/raft/blob/v1.8.0/file_snapshot.go#L458-L464)). Reaping skips `.tmp` dirs, so these pile up with each failed attempt.

Raft calls `Cancel()` on exactly these paths, e.g. when `Persist` fails ([snapshot.go#L192](https://github.com/hashicorp/raft/blob/v1.8.0/snapshot.go#L192)). #224 fixed the directory cleanup for `Close()`, but `Cancel()` was missed.

**Repro** (Linux, v1.8.0): this caps file size so writes past 1 MiB fail, then counts what's left open.

```go
//go:build linux

// Reproduces FileSnapshotSink cleanup problems after a failed state file write.
// RLIMIT_FSIZE makes writes past 1 MiB fail with EFBIG, standing in for a full disk.
// GC is disabled so os.File finalizers don't close leaked files during the run.
package main

import (
"fmt"
"os"
"os/signal"
"path/filepath"
"runtime/debug"
"strings"
"syscall"

"github.com/hashicorp/raft"
)

// openStateFiles counts open file descriptors pointing at a state.bin under dir.
func openStateFiles(dir string) int {
entries, err := os.ReadDir("/proc/self/fd")
if err != nil {
panic(err)
}
n := 0
for _, e := range entries {
target, err := os.Readlink(filepath.Join("/proc/self/fd", e.Name()))
if err == nil && strings.HasPrefix(target, dir) && strings.Contains(target, "state.bin") {
n++
}
}
return n
}

func run(method string) {
dir, err := os.MkdirTemp("", "raft-snapshot-repro")
if err != nil {
panic(err)
}
defer os.RemoveAll(dir)

store, err := raft.NewFileSnapshotStore(dir, 1, os.Stderr)
if err != nil {
panic(err)
}
_, trans := raft.NewInmemTransport("")

sink, err := store.Create(1, 10, 1, raft.Configuration{}, 0, trans)
if err != nil {
panic(err)
}
_, writeErr := sink.Write(make([]byte, 2<<20))

var endErr error
if method == "Cancel" {
endErr = sink.Cancel()
} else {
endErr = sink.Close()
}

tmpDirs, _ := filepath.Glob(filepath.Join(dir, "snapshots", "*.tmp"))
fmt.Printf("%s: write failed=%t, %s returned error=%t, state.bin still open=%d, .tmp dirs left=%d\n",
method, writeErr != nil, method, endErr != nil, openStateFiles(dir), len(tmpDirs))
}

func main() {
debug.SetGCPercent(-1)
signal.Ignore(syscall.SIGXFSZ)
var lim syscall.Rlimit
if err := syscall.Getrlimit(syscall.RLIMIT_FSIZE, &lim); err != nil {
panic(err)
}
lim.Cur = 1 << 20
if err := syscall.Setrlimit(syscall.RLIMIT_FSIZE, &lim); err != nil {
panic(err)
}

run("Cancel")
run("Close")
}
```

```
$ go run . 2>/dev/null
Cancel: write failed=true, Cancel returned error=true, state.bin still open=1, .tmp dirs left=1
Close: write failed=true, Close returned error=true, state.bin still open=1, .tmp dirs left=0
```

Both should show `state.bin still open=0` and `.tmp dirs left=0`.

The fix is small: close `stateFile` on the error paths in `finalize()`, and add the same `RemoveAll` cleanup to `Cancel()`. I have it ready with a regression test and can open a PR if that approach works for you.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in file_snapshot.go at finalize() and Cancel(), then run the Linux RLIMIT_FSIZE reproduction described in the issue. Add the regression test mentioned in the report and verify that failed writes leave state.bin closed and remove all .tmp directories for both Cancel() and Close().

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.