hashicorp / hashicorp/consul

1.16.3 panic when call NodeDump parseNodes :fatal error: concurrent map writes

Open
#20,206 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

1.16.3 panic when call NodeDump parseNodes :fatal error: concurrent map writes

---

#### Reproduction Steps

#### panic stack
`
Jan 11 16:51:13 consul-client: fatal error: concurrent map writes
Jan 11 16:51:13 consul-client: goroutine 115186222 [running]:
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/consul/state.parseNodes({0x4e2b570, 0xc00ffefd10}, 0xc001705680, 0x40b9e88?, {0x4e20848, 0xc015063c10}, 0xc000d1c680?, {0x0?, 0x0?})
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/consul/state/catalog.go:3471 +0xccc
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/consul/state.(*Store).NodeDump(0xc000d27cc0?, 0xc001705680, 0xc00ceccbd8, {0x0, 0x0})
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/consul/state/catalog.go:3345 +0x33f
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/consul.(*Internal).NodeDump.func1(0x3cd36e0?, 0xc001705680?)
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/consul/internal_endpoint.go:86 +0x91
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/blockingquery.Query({0x4e433f0, 0xc000a20400}, {0x4e3a078, 0xc00ceccbd8}, {0x4e3c760, 0xc00b200180}, 0xc011b294c0)
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/blockingquery/blockingquery.go:171 +0x5c4
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/consul.(*Server).blockingQuery(...)
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/consul/rpc.go:1029
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/consul.(*Internal).NodeDump(0xc000bd6660, 0xc00ceccb60, 0xc00b200150)
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/consul/internal_endpoint.go:74 +0x156
Jan 11 16:51:13 consul-client: reflect.Value.call({0xc000be04e0?, 0xc0001352d8?, 0xc01000b000?}, {0x40af1b7, 0x4}, {0xc011b29c90, 0x3, 0xc00ceccb60?})
Jan 11 16:51:13 consul-client: reflect/value.go:586 +0xb07
Jan 11 16:51:13 consul-client: reflect.Value.Call({0xc000be04e0?, 0xc0001352d8?, 0xb57048?}, {0xc011b29c90?, 0x3ef3260?, 0xc011b29c58?})
Jan 11 16:51:13 consul-client: reflect/value.go:370 +0xbc
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul-net-rpc/net/rpc.(*service).call(0xc000bcd180, 0x403ffa0?, 0x3dd8920?, 0x0, 0xc0000e9100, 0x0?, {0x403ffa0?, 0xc00ceccb60?, 0x4d6e37?}, {0x3dd8920, ...}, ...)
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul-net-rpc@v0.0.0-20221205195236-156cfab66a69/net/rpc/server.go:409 +0x22e
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul-net-rpc/net/rpc.(*Server).ServeRequest.func1()
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul-net-rpc@v0.0.0-20221205195236-156cfab66a69/net/rpc/server.go:490 +0x70
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/rpc/middleware.GetNetRPCInterceptor.func1({0xc00294c768, 0x11}, {0x403ffa0?, 0xc00ceccb60?, 0x4e3c4a0?}, {0xc00e147540?, 0x1ff55c5?, 0x3?}, 0xc00b2001c0)
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/rpc/middleware/interceptors.go:160 +0x90
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul-net-rpc/net/rpc.(*Server).ServeRequest(0xc0008b6320, {0x4e3c4a0?, 0xc00e147540})
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul-net-rpc@v0.0.0-20221205195236-156cfab66a69/net/rpc/server.go:494 +0x38d
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/consul.(*Server).handleConsulConn(0xc000a20400, {0x4e45a50?, 0xc00285bad0})
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/consul/rpc.go:430 +0x148
Jan 11 16:51:13 consul-client: created by github.com/hashicorp/consul/agent/consul.(*Server).handleMultiplexV2
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/agent/consul/rpc.go:414 +0x37e
Jan 11 16:51:13 consul-client: goroutine 1 [select, 4224 minutes]:
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/command/agent.(*cmd).run(0xc000ac7000, {0xc0000720a0, 0x2, 0x2})
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/command/agent/agent.go:276 +0x1b47
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/command/agent.(*cmd).Run(0xc000ac7000, {0xc0000720a0?, 0xffffffffffffffff?, 0x0?})
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/command/agent/agent.go:74 +0x27
Jan 11 16:51:13 consul-client: github.com/mitchellh/cli.(*CLI).Run(0xc000d27180)
Jan 11 16:51:13 consul-client: github.com/mitchellh/cli@v1.1.4/cli.go:262 +0x5f8
Jan 11 16:51:13 consul-client: main.realMain()
Jan 11 16:51:13 consul-client: github.com/hashicorp/consul/main.go:51 +0x40f
Jan 11 16:51:13 consul-client: main.main()

`

Contributor guide

Open the contributing guide

Research direction

Start with agent/consul/state/catalog.go at NodeDump and parseNodes, especially the reported lines 3345 and 3471, then follow the NodeDump entry point in agent/consul/internal_endpoint.go at lines 74-86. Investigate the concurrent map write shown in the stack trace and confirm that concurrent NodeDump requests no longer cause a fatal panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.