hashicorp / hashicorp/consul

Sporadic error when saving snapshots: unexpected EOF

Open
#6,893 10 comments 2 reactions 0 assignees View on GitHub
theme/internals type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.

#### Overview of the Issue

I keep getting random errors when creating backups of our Consul.

$ consul snapshot save /consul/data/20191206120000-1.snap
Error verifying snapshot file: failed to read snapshot file: failed to read or write snapshot data: unexpected EOF

#### Reproduction Steps

Just keep creating snapshots and you can see that it sometimes fails.

$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068184
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068185
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068186
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068186
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068188
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068188
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068188
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068188
$ consul snapshot save /consul/data/20191206120000-1.snap
Error verifying snapshot file: failed to read snapshot file: failed to read or write snapshot data: unexpected EOF
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068194
$ consul snapshot save /consul/data/20191206120000-1.snap
Saved and verified snapshot to index 1068195

### Consul info for both Client and Server

$ consul version
Consul v1.6.1
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Client info

```
output from client 'consul info' command here
```

Server info

```
agent:
check_monitors = 0
check_ttls = 3
checks = 3
services = 3
build:
prerelease =
revision = 9be6dfc3
version = 1.6.1
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.129.8.88:8300
server = true
raft:
applied_index = 1068318
commit_index = 1068318
fsm_pending = 0
last_contact = 10.52169ms
last_log_index = 1068318
last_log_term = 332
last_snapshot_index = 1061622
last_snapshot_term = 332
latest_configuration = [{Suffrage:Voter ID:faea4c4e-2b87-83af-b831-9b3fe9b34c58 Address:10.131.12.242:8300} {Suffrage:Voter ID:a1b1991e-8f46-45b6-a09f-3c3cd1155bba Address:10.130.13.199:8300} {Suffrage:Voter ID:3261cc81-307f-c0be-3e7a-e224704bf2a3 Address:10.129.8.88:8300} {Suffrage:Voter ID:a12d5052-dc73-b1af-ed86-313e6c0d038c Address:10.130.6.140:8300} {Suffrage:Voter ID:14e82d1e-a6f2-eba8-d2ff-cc648a5daaac Address:10.129.12.38:8300}]
latest_configuration_index = 1047983
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 332
runtime:
arch = amd64
cpu_count = 4
goroutines = 92
max_procs = 4
os = linux
version = go1.12.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 19
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 137
members = 5
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 193
members = 5
query_queue = 0
query_time = 1
```

### Operating system and Environment details

OS, Architecture, and any other information you can provide about the environment.

### Log Fragments

Include appropriate Client or Server log fragments. If the log is longer than a few dozen lines, please include the URL to the [gist](https://gist.github.com/) of the log instead of posting it in the issue. Use `-log-level=TRACE` on the client and server to capture the maximum log detail.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.