hashicorp / hashicorp/nomad

Connection for Connect causes Consul confusion on closing

Open
#10,846 0 comments 4 reactions 0 assignees View on GitHub
hcc/cst stage/accepted theme/consul/connect type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

When shutting down a task with a Connect sidecar, Nomad causes an error like this in Consul

```
2021-07-02T10:15:00.506-0500 [ERROR] agent.envoy: Error receiving new DeltaDiscoveryRequest; closing request channel: error="rpc error: code = Canceled desc = context canceled"
```

I suspect this has something to do with re-using a Context with `Context.Cancel` method being called to stop the connect which causes an error to be written over the wire to Consul.

You can see here it's impossible to call `Close` on the connection without the context already being cancelled. Since that context is shared when calling `DialContext`, I suspect the context cancellation error is being propagated over the wire.
https://github.com/hashicorp/nomad/blob/main/client/allocrunner/consul_grpc_sock_hook.go#L347

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in client/allocrunner/consul_grpc_sock_hook.go around line 347 and trace how the shared context is passed to DialContext and then cancelled during Connect sidecar shutdown. Reproduce the shutdown and inspect the connection close path; done when closing the connection no longer sends the context-cancellation error to Consul.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.