Sporadic error when saving snapshots: unexpected EOF
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.
#### Overview of the Issue
I keep getting random errors when creating backups of our Consul.
$ consul snapshot save /consul/data/20191206120000-1.snap
Error verifying snapshot file: failed to read snapshot file: failed to read or write snapshot data: unexpected EOF
#### Reproduction Steps
Just keep creating snapshots and you can see that it sometimes fails.
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068184
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068185
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068186
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068186
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068188
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068188
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068188
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068188
$ consul snapshot save /consul/data/20191206120000-1.snap
Error verifying snapshot file: failed to read snapshot file: failed to read or write snapshot data: unexpected EOF
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068194
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068195
### Consul info for both Client and Server
$ consul version
Consul v1.6.1
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
Client info
```
output from client 'consul info' command here
```
Server info
```
agent:
check_monitors = 0
check_ttls = 3
checks = 3
services = 3
build:
prerelease =
revision = 9be6dfc3
version = 1.6.1
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.129.8.88:8300
server = true
raft:
applied_index = 1068318
commit_index = 1068318
fsm_pending = 0
last_contact = 10.52169ms
last_log_index = 1068318
last_log_term = 332
last_snapshot_index = 1061622
last_snapshot_term = 332
latest_configuration = [{Suffrage:Voter ID:faea4c4e-2b87-83af-b831-9b3fe9b34c58 Address:10.131.12.242:8300} {Suffrage:Voter ID:a1b1991e-8f46-45b6-a09f-3c3cd1155bba Address:10.130.13.199:8300} {Suffrage:Voter ID:3261cc81-307f-c0be-3e7a-e224704bf2a3 Address:10.129.8.88:8300} {Suffrage:Voter ID:a12d5052-dc73-b1af-ed86-313e6c0d038c Address:10.130.6.140:8300} {Suffrage:Voter ID:14e82d1e-a6f2-eba8-d2ff-cc648a5daaac Address:10.129.12.38:8300}]
latest_configuration_index = 1047983
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 332
runtime:
arch = amd64
cpu_count = 4
goroutines = 92
max_procs = 4
os = linux
version = go1.12.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 19
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 137
members = 5
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 193
members = 5
query_queue = 0
query_time = 1
```
### Operating system and Environment details
OS, Architecture, and any other information you can provide about the environment.
### Log Fragments
Include appropriate Client or Server log fragments. If the log is longer than a few dozen lines, please include the URL to the [gist](https://gist.github.com/) of the log instead of posting it in the issue. Use `-log-level=TRACE` on the client and server to capture the maximum log detail.
Contributor guide
Assessment
This issue has not been assessed yet.