v2.0.4 giving i/o deadline reached errors
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
After upgrading for v2.0.3 to v2.0.4 on a small cluster, the servers were getting errors about i/o deadline reached:
2026-09-12T11:56:16.991-0400 [ERROR] agent.server.rpc: RPC error: conn=from=10.5.15.213:44803 error="rpc: server cannot decode request: i/o deadline reached"
Clients were reporting these errors:
Sep 12 11:56:35 consul-client.dev.dimins.com consul[1873]: 2026-09-12T11:56:35.072-0400 [ERROR] agent.client: RPC failed to server: method=Coordinate.Update server=10.5.15.212:8300 error="rpc error making call: EOF"
Sep 12 11:56:35 consul-client.dev.dimins.com consul[1873]: 2026-09-12T11:56:35.072-0400 [ERROR] agent: Coordinate update error: error="rpc error making call: EOF"
Sep 12 11:56:35 consul-client.dev.dimins.com consul[1873]: agent.client: RPC failed to server: method=Coordinate.Update server=10.5.15.212:8300 error="rpc error making call: EOF"
Sep 12 11:56:35 consul-client.dev.dimins.com consul[1873]: agent: Coordinate update error: error="rpc error making call: EOF"
...
Sep 12 12:01:31 consul-client.dev.dimins.com consul[1873]: agent.client: Retrying RPC to server: method=Catalog.NodeServiceList server=10.5.15.212:8300 error="rpc error making call: EOF"
---
#### Reproduction Steps
I created a test bed with just one server and one client, and compiled different versions for the server between 2.0.3 and 2.0.4 to bisect the issue. commit 22c5407203 shows the errors, previous commit 230e22ccba does not.
Client had briefly joined the bigger cluster earlier, but did remove /var/cache/consul/serf before joining test cluster.
I tried increasing rpc_handshake_timeout, but that did not solve the problem.
As this is a testbed running off of source, I can easily run test code.
### Consul info for both Client and Server
Client info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 3
build:
prerelease =
revision = a3eb721c
version = 2.0.4
version_metadata =
consul:
acl = disabled
known_servers = 1
server = false
runtime:
arch = amd64
cpu_count = 32
goroutines = 48
max_procs = 2
os = linux
version = go1.26.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 9
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 262680
members = 2
query_queue = 0
query_time = 1
```
```
{
"datacenter": "markley",
"bind_addr": "{{ GetInterfaceIP \"eth0\" }}",
"data_dir": "/var/cache/consul",
"encrypt": "",
"log_level": "INFO",
"disable_update_check": true,
"enable_syslog": true,
"retry_join": [
"consul-server.dev.dimins.com"
]
}
```
Server info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease = dev
revision = 22c54072
version = 2.0.2
version_metadata =
consul:
acl = disabled
bootstrap = true
known_datacenters = 1
leader = true
leader_addr = 10.5.15.212:8300
server = true
raft:
applied_index = 262
commit_index = 262
fsm_pending = 0
last_contact = 0
last_log_index = 262
last_log_term = 9
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:4f0999e0-4c72-0a51-7ddb-c077545363b2 Address:10.5.15.212:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 9
runtime:
arch = amd64
cpu_count = 32
goroutines = 192
max_procs = 32
os = linux
version = go1.27.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 9
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 262680
members = 2
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```
```
datacenter = "markley"
data_dir = "/data/consul"
encrypt = ""
disable_update_check = true
server = true
bootstrap_expect = 1
```
### Operating system and Environment details
Debian bookworm
### Log Fragments
Server at TRACE:
2026-09-12T12:14:21.871-0400 [TRACE] agent.server: rpc_server_call: method=Coordinate.Update errored=false request_type=write rpc_type=net/rpc leader=true target_datacenter=markley locality=local
2026-09-12T12:14:22.637-0400 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2026-09-12T12:14:24.634-0400 [TRACE] agent.server.usage_metrics: Starting usage run
2026-09-12T12:14:24.636-0400 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2026-09-12T12:14:26.637-0400 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2026-09-12T12:14:26.718-0400 [ERROR] agent.server.rpc: RPC error: conn=from=10.5.15.213:59507 error="rpc: server cannot decode request: i/o deadline reached"
Client at TRACE:
Sep 12 12:17:07 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:07.272-0400 [DEBUG] agent.client.memberlist.lan: memberlist: Initiating push/pull sync with: consul-server.dev.dimins.com 10.5.15.212:8301
Sep 12 12:17:07 consul-client.dev.dimins.com consul[3767]: agent.client.memberlist.lan: memberlist: Initiating push/pull sync with: consul-server.dev.dimins.com 10.5.15.212:8301
Sep 12 12:17:20 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:20.442-0400 [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Sep 12 12:17:20 consul-client.dev.dimins.com consul[3767]: agent.proxycfg.agent-state: syncing proxy services from local state
Sep 12 12:17:21 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:21.911-0400 [DEBUG] agent.client.memberlist.lan: memberlist: Stream connection from=10.5.15.212:46248
Sep 12 12:17:21 consul-client.dev.dimins.com consul[3767]: agent.client.memberlist.lan: memberlist: Stream connection from=10.5.15.212:46248
Sep 12 12:17:25 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:25.073-0400 [ERROR] agent.client: RPC failed to server: method=Coordinate.Update server=10.5.15.212:8300 error="rpc error making call: EOF"
Sep 12 12:17:25 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:25.073-0400 [ERROR] agent: Coordinate update error: error="rpc error making call: EOF"
Sep 12 12:17:25 consul-client.dev.dimins.com consul[3767]: agent.client: RPC failed to server: method=Coordinate.Update server=10.5.15.212:8300 error="rpc error making call: EOF"
Sep 12 12:17:25 consul-client.dev.dimins.com consul[3767]: agent: Coordinate update error: error="rpc error making call: EOF"
Contributor guide
Research direction
Start by reproducing the one-server/one-client setup and bisecting between commits 230e22ccba and 22c5407203. Inspect the RPC paths for Coordinate.Update and request decoding, using the supplied server and client TRACE logs as checkpoints. Done means identifying why the regression causes i/o deadline reached and EOF errors and demonstrating that the errors no longer occur.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100