Connection for Connect causes Consul confusion on closing
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
When shutting down a task with a Connect sidecar, Nomad causes an error like this in Consul
```
2021-07-02T10:15:00.506-0500 [ERROR] agent.envoy: Error receiving new DeltaDiscoveryRequest; closing request channel: error="rpc error: code = Canceled desc = context canceled"
```
I suspect this has something to do with re-using a Context with `Context.Cancel` method being called to stop the connect which causes an error to be written over the wire to Consul.
You can see here it's impossible to call `Close` on the connection without the context already being cancelled. Since that context is shared when calling `DialContext`, I suspect the context cancellation error is being propagated over the wire.
https://github.com/hashicorp/nomad/blob/main/client/allocrunner/consul_grpc_sock_hook.go#L347
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in client/allocrunner/consul_grpc_sock_hook.go around line 347 and trace how the shared context is passed to DialContext and then cancelled during Connect sidecar shutdown. Reproduce the shutdown and inspect the connection close path; done when closing the connection no longer sends the context-cancellation error to Consul.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grpc
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100