hashicorp / hashicorp/consul

Consul doesn't clean up raft snapshot directory on error

Open
#5,314 3 comments 4 reactions 0 assignees View on GitHub
help-wanted type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

Specifically in this case, when the filesystem consul/data/raft/snapshots sits on runs out of space, Consul will not delete the .tmp directory.

#### Reproduction Steps

Let the snapshot run while there is no space on the volume the snapshots are saved to.

### Consul info for both Client and Server

```
agent:
check_monitors = 0
check_ttls = 0
checks = 3
services = 3
build:
prerelease =
revision = e716d1b5
version = 1.2.2
consul:
bootstrap = false
known_datacenters = 2
leader = false
leader_addr = 10.10.17.81:8300
server = true
raft:
applied_index = 1364683361
commit_index = 1364683361
fsm_pending = 0
last_contact = 20.351898ms
last_log_index = 1364683362
last_log_term = 12843
last_snapshot_index = 1364669339
last_snapshot_term = 12843
latest_configuration = [{Suffrage:Voter ID:d23843a2-f48f-b710-7c7d-d9b426a3c917 Address:10.10.15.166:8300} {Suffrage:Voter ID:f94d667f-a36b-dfd7-1398-ab33d14c6b2a Address:10.10.0.72:8300} {Suffrage:Voter ID:c7360714-0cba-4289-9654-460fed891575 Address:10.10.0.51:8300} {Suffrage:Voter ID:470b7925-ecff-df67-5b8b-b530b1d9478e Address:10.10.17.81:8300} {Suffrage:Voter ID:15783a8e-6f51-38d7-b1d1-9710d6ff156f Address:10.10.15.93:8300}]
latest_configuration_index = 1113600355
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 12843
runtime:
arch = amd64
cpu_count = 4
goroutines = 938
max_procs = 5
os = linux
version = go1.10.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 22754
failed = 0
health_score = 0
intent_queue = 0
left = 11
member_time = 163888
members = 452
query_queue = 0
query_time = 25
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5720
members = 8
query_queue = 0
query_time = 2
```

### Operating system and Environment details

CentOS 7 EC2 instance.

### Log Fragments
```
2019/02/04 16:01:58 [INFO] consul.fsm: snapshot created in 46.652µs
2019/02/04 16:01:58 [INFO] raft: Starting snapshot up to 1364666899
2019/02/04 16:01:58 [INFO] snapshot: Creating new snapshot at /opt/consul/data/raft/snapshots/12843-1364666899-1549314118552.tmp
2019/02/04 16:02:00 [ERR] snapshot: Failed to finalize snapshot: write /opt/consul/data/raft/snapshots/12843-1364666899-1549314118552.tmp/state.bin: no space left on device
2019/02/04 16:02:00 [ERR] raft: Failed to take snapshot: failed to persist snapshot: write /opt/consul/data/raft/snapshots/12843-1364666899-1549314118552.tmp/state.bin: no space left on device
```
When that process starts there is enough space, but then it runs our during the above and fails w/the error you see. Maddeningly, only one server in the cluster is (a follower) is having this issue despite them being identical. Also, it doesn't happen every time either.

However, the thrust of this bug report is that when the above happens, the .tmp snapshot directory is left, thus ensuring the next will fail. When the snapshot fails to persists (or at all, really) it should clean up the temporary directory it made.

Contributor guide

Open the contributing guide

Research direction

Start with the snapshot creation and finalization path identified by the log messages, especially handling of the `.tmp` directory under consul/data/raft/snapshots. Reproduce the no-space condition and verify that a failed snapshot removes its temporary directory instead of leaving it behind.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.