hashicorp / hashicorp/consul

v2.0.4 giving i/o deadline reached errors

Open
#23,923 8 comments 2 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

After upgrading for v2.0.3 to v2.0.4 on a small cluster, the servers were getting errors about i/o deadline reached:

2026-09-12T11:56:16.991-0400 [ERROR] agent.server.rpc: RPC error: conn=from=10.5.15.213:44803 error="rpc: server cannot decode request: i/o deadline reached"

Clients were reporting these errors:

Sep 12 11:56:35 consul-client.dev.dimins.com consul[1873]: 2026-09-12T11:56:35.072-0400 [ERROR] agent.client: RPC failed to server: method=Coordinate.Update server=10.5.15.212:8300 error="rpc error making call: EOF"
Sep 12 11:56:35 consul-client.dev.dimins.com consul[1873]: 2026-09-12T11:56:35.072-0400 [ERROR] agent: Coordinate update error: error="rpc error making call: EOF"
Sep 12 11:56:35 consul-client.dev.dimins.com consul[1873]: agent.client: RPC failed to server: method=Coordinate.Update server=10.5.15.212:8300 error="rpc error making call: EOF"
Sep 12 11:56:35 consul-client.dev.dimins.com consul[1873]: agent: Coordinate update error: error="rpc error making call: EOF"
...
Sep 12 12:01:31 consul-client.dev.dimins.com consul[1873]: agent.client: Retrying RPC to server: method=Catalog.NodeServiceList server=10.5.15.212:8300 error="rpc error making call: EOF"

---

#### Reproduction Steps

I created a test bed with just one server and one client, and compiled different versions for the server between 2.0.3 and 2.0.4 to bisect the issue. commit 22c5407203 shows the errors, previous commit 230e22ccba does not.

Client had briefly joined the bigger cluster earlier, but did remove /var/cache/consul/serf before joining test cluster.

I tried increasing rpc_handshake_timeout, but that did not solve the problem.

As this is a testbed running off of source, I can easily run test code.

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 3
build:
prerelease =
revision = a3eb721c
version = 2.0.4
version_metadata =
consul:
acl = disabled
known_servers = 1
server = false
runtime:
arch = amd64
cpu_count = 32
goroutines = 48
max_procs = 2
os = linux
version = go1.26.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 9
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 262680
members = 2
query_queue = 0
query_time = 1

```

```
{
"datacenter": "markley",
"bind_addr": "{{ GetInterfaceIP \"eth0\" }}",
"data_dir": "/var/cache/consul",
"encrypt": "",
"log_level": "INFO",
"disable_update_check": true,
"enable_syslog": true,
"retry_join": [
"consul-server.dev.dimins.com"
]
}
```

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease = dev
revision = 22c54072
version = 2.0.2
version_metadata =
consul:
acl = disabled
bootstrap = true
known_datacenters = 1
leader = true
leader_addr = 10.5.15.212:8300
server = true
raft:
applied_index = 262
commit_index = 262
fsm_pending = 0
last_contact = 0
last_log_index = 262
last_log_term = 9
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:4f0999e0-4c72-0a51-7ddb-c077545363b2 Address:10.5.15.212:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 9
runtime:
arch = amd64
cpu_count = 32
goroutines = 192
max_procs = 32
os = linux
version = go1.27.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 9
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 262680
members = 2
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```

```
datacenter = "markley"
data_dir = "/data/consul"
encrypt = ""

disable_update_check = true

server = true

bootstrap_expect = 1
```

### Operating system and Environment details

Debian bookworm

### Log Fragments

Server at TRACE:
2026-09-12T12:14:21.871-0400 [TRACE] agent.server: rpc_server_call: method=Coordinate.Update errored=false request_type=write rpc_type=net/rpc leader=true target_datacenter=markley locality=local
2026-09-12T12:14:22.637-0400 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2026-09-12T12:14:24.634-0400 [TRACE] agent.server.usage_metrics: Starting usage run
2026-09-12T12:14:24.636-0400 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2026-09-12T12:14:26.637-0400 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2026-09-12T12:14:26.718-0400 [ERROR] agent.server.rpc: RPC error: conn=from=10.5.15.213:59507 error="rpc: server cannot decode request: i/o deadline reached"

Client at TRACE:
Sep 12 12:17:07 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:07.272-0400 [DEBUG] agent.client.memberlist.lan: memberlist: Initiating push/pull sync with: consul-server.dev.dimins.com 10.5.15.212:8301
Sep 12 12:17:07 consul-client.dev.dimins.com consul[3767]: agent.client.memberlist.lan: memberlist: Initiating push/pull sync with: consul-server.dev.dimins.com 10.5.15.212:8301
Sep 12 12:17:20 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:20.442-0400 [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Sep 12 12:17:20 consul-client.dev.dimins.com consul[3767]: agent.proxycfg.agent-state: syncing proxy services from local state
Sep 12 12:17:21 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:21.911-0400 [DEBUG] agent.client.memberlist.lan: memberlist: Stream connection from=10.5.15.212:46248
Sep 12 12:17:21 consul-client.dev.dimins.com consul[3767]: agent.client.memberlist.lan: memberlist: Stream connection from=10.5.15.212:46248
Sep 12 12:17:25 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:25.073-0400 [ERROR] agent.client: RPC failed to server: method=Coordinate.Update server=10.5.15.212:8300 error="rpc error making call: EOF"
Sep 12 12:17:25 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:25.073-0400 [ERROR] agent: Coordinate update error: error="rpc error making call: EOF"
Sep 12 12:17:25 consul-client.dev.dimins.com consul[3767]: agent.client: RPC failed to server: method=Coordinate.Update server=10.5.15.212:8300 error="rpc error making call: EOF"
Sep 12 12:17:25 consul-client.dev.dimins.com consul[3767]: agent: Coordinate update error: error="rpc error making call: EOF"

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the one-server/one-client setup and bisecting between commits 230e22ccba and 22c5407203. Inspect the RPC paths for Coordinate.Update and request decoding, using the supplied server and client TRACE logs as checkpoints. Done means identifying why the regression causes i/o deadline reached and EOF errors and demonstrating that the errors no longer occur.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.