hashicorp / hashicorp/consul

Terminating Gateway Routing to SM Service Instances

Open
#20,166 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

When a service has some instances that are on mesh, running the sidecar proxy, and other instances that are external, Terminating Gateway is occasionally attempting to route to a service mesh instance of the service. Both services are otherwise healthy and responding as expected when:
a). terminating gateway routes to the externally registered instances
b). on-mesh instances route directly

The response on the downstream when it fails is - `upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111`

Possibly related to most recent comment of PR https://github.com/hashicorp/consul/pull/13958.

---

#### Reproduction Steps

1. Create a cluster
1. Deploy service mesh instances of dashboard and counting
1. Deploy and register external counting service instance, registering with the same service name
1. Deploy terminating gateway with counting service configured
1. Enable envoy debug logging on terminating gateway
1. Create service intentions
1. From upstream/dashboard curl local port binding for counting upstream until error is thrown

### Operating system and Environment details

Issue discovered in Nomad 1.6.x environment. Reproduced on generic VMs.
Consul 1.16.3

### Log Fragments

Terminating Gateway Logs
```log
[2024-01-11 04:34:40.897][91][debug][pool] [source/common/http/conn_pool_base.cc:78] queueing stream due to no available connections (ready=0 busy=0 connecting=0)
[2024-01-11 04:34:40.897][91][debug][pool] [source/common/conn_pool/conn_pool_base.cc:291] trying to create new connection
[2024-01-11 04:34:40.897][91][debug][pool] [source/common/conn_pool/conn_pool_base.cc:145] creating a new connection (connecting=0)
[2024-01-11 04:34:40.897][91][debug][connection] [./source/common/network/connection_impl.h:98] [C59] current connecting state: true
[2024-01-11 04:34:40.897][91][debug][client] [source/common/http/codec_client.cc:57] [C59] connecting
[2024-01-11 04:34:40.897][91][debug][connection] [source/common/network/connection_impl.cc:941] [C59] connecting to 172.20.0.11:9001
[2024-01-11 04:34:40.897][91][debug][connection] [source/common/network/connection_impl.cc:960] [C59] connection in progress
[2024-01-11 04:34:42.567][40][debug][main] [source/server/server.cc:265] flushing stats
[2024-01-11 04:34:45.897][91][debug][pool] [source/common/conn_pool/conn_pool_base.cc:793] [C59] connect timeout
[2024-01-11 04:34:45.897][91][debug][connection] [source/common/network/connection_impl.cc:139] [C59] closing data_to_write=0 type=1
[2024-01-11 04:34:45.897][91][debug][connection] [source/common/network/connection_impl.cc:250] [C59] closing socket: 1
[2024-01-11 04:34:45.897][91][debug][client] [source/common/http/codec_client.cc:107] [C59] disconnect. resetting 0 pending requests
[2024-01-11 04:34:45.897][91][debug][pool] [source/common/conn_pool/conn_pool_base.cc:484] [C59] client disconnected, failure reason:
[2024-01-11 04:34:45.897][91][debug][router] [source/common/router/router.cc:1278] [C38][S13396964795188056224] upstream reset: reset reason: connection failure, transport failure reason:
[2024-01-11 04:34:45.898][91][debug][http] [source/common/http/filter_manager.cc:996] [C38][S13396964795188056224] Sending local reply with details upstream_reset_before_response_started{connection_failure}
[2024-01-11 04:34:45.898][91][debug][http] [source/common/http/conn_manager_impl.cc:1773] [C38][S13396964795188056224] encoding headers via codec (end_stream=false):
':status', '503'
'content-length', '91'
'content-type', 'text/plain'
'date', 'Thu, 11 Jan 2024 04:34:45 GMT'
'server', 'envoy'

[2024-01-11 04:34:45.898][91][debug][http] [source/common/http/conn_manager_impl.cc:1865] [C38][S13396964795188056224] Codec completed encoding stream.
[2024-01-11 04:34:45.898][91][debug][pool] [source/common/conn_pool/conn_pool_base.cc:454] invoking idle callbacks - is_draining_for_deletion_=false
```

where:
```
instance | ip | port
counting-1 | 172.20.0.11 | 9001 (local_app)
counting-external-1 | 172.20.0.6 | 9999
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.