Consul doesn't clean up raft snapshot directory on error
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
Specifically in this case, when the filesystem consul/data/raft/snapshots sits on runs out of space, Consul will not delete the .tmp directory.
#### Reproduction Steps
Let the snapshot run while there is no space on the volume the snapshots are saved to.
### Consul info for both Client and Server
```
agent:
check_monitors = 0
check_ttls = 0
checks = 3
services = 3
build:
prerelease =
revision = e716d1b5
version = 1.2.2
consul:
bootstrap = false
known_datacenters = 2
leader = false
leader_addr = 10.10.17.81:8300
server = true
raft:
applied_index = 1364683361
commit_index = 1364683361
fsm_pending = 0
last_contact = 20.351898ms
last_log_index = 1364683362
last_log_term = 12843
last_snapshot_index = 1364669339
last_snapshot_term = 12843
latest_configuration = [{Suffrage:Voter ID:d23843a2-f48f-b710-7c7d-d9b426a3c917 Address:10.10.15.166:8300} {Suffrage:Voter ID:f94d667f-a36b-dfd7-1398-ab33d14c6b2a Address:10.10.0.72:8300} {Suffrage:Voter ID:c7360714-0cba-4289-9654-460fed891575 Address:10.10.0.51:8300} {Suffrage:Voter ID:470b7925-ecff-df67-5b8b-b530b1d9478e Address:10.10.17.81:8300} {Suffrage:Voter ID:15783a8e-6f51-38d7-b1d1-9710d6ff156f Address:10.10.15.93:8300}]
latest_configuration_index = 1113600355
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 12843
runtime:
arch = amd64
cpu_count = 4
goroutines = 938
max_procs = 5
os = linux
version = go1.10.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 22754
failed = 0
health_score = 0
intent_queue = 0
left = 11
member_time = 163888
members = 452
query_queue = 0
query_time = 25
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5720
members = 8
query_queue = 0
query_time = 2
```
### Operating system and Environment details
CentOS 7 EC2 instance.
### Log Fragments
```
2019/02/04 16:01:58 [INFO] consul.fsm: snapshot created in 46.652µs
2019/02/04 16:01:58 [INFO] raft: Starting snapshot up to 1364666899
2019/02/04 16:01:58 [INFO] snapshot: Creating new snapshot at /opt/consul/data/raft/snapshots/12843-1364666899-1549314118552.tmp
2019/02/04 16:02:00 [ERR] snapshot: Failed to finalize snapshot: write /opt/consul/data/raft/snapshots/12843-1364666899-1549314118552.tmp/state.bin: no space left on device
2019/02/04 16:02:00 [ERR] raft: Failed to take snapshot: failed to persist snapshot: write /opt/consul/data/raft/snapshots/12843-1364666899-1549314118552.tmp/state.bin: no space left on device
```
When that process starts there is enough space, but then it runs our during the above and fails w/the error you see. Maddeningly, only one server in the cluster is (a follower) is having this issue despite them being identical. Also, it doesn't happen every time either.
However, the thrust of this bug report is that when the above happens, the .tmp snapshot directory is left, thus ensuring the next will fail. When the snapshot fails to persists (or at all, really) it should clean up the temporary directory it made.
Contributor guide
Research direction
Start with the snapshot creation and finalization path identified by the log messages, especially handling of the `.tmp` directory under consul/data/raft/snapshots. Reproduce the no-space condition and verify that a failed snapshot removes its temporary directory instead of leaving it behind.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100