hashicorp / hashicorp/consul

RandomSampling on ingress gateway is suppressing distributed tracing

Open
#8,519 14 comments 5 reactions 0 assignees View on GitHub
theme/connect type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

Ingress Gateway listener requires use of Envoy's `x-client-trace-id` header to initiate the trace. Without this header the requests are not traced at all.

#### Reproduction Steps

Create a new ingress gateway with tracing configuration. This example uses datadog tracer:

Ingress Gateway Service Config

```
service {
name = "igw"
port = 9999
kind = "ingress-gateway"
proxy {
config {
envoy_dogstatsd_url = "udp://127.0.0.1:8125"
envoy_tracing_json = <<-EOF
{
"http": {
"name": "envoy.tracers.datadog",
"config": {
"collector_cluster": "datadog_trace_collector",
"service_name": "igw"
}
}
}
EOF
envoy_extra_static_clusters_json = <<-EOF
{
"name": "datadog_trace_collector",
"type": "STATIC",
"connect_timeout": "1s",
"upstream_connection_options": {
"tcp_keepalive": {}
},
"load_assignment": {
"cluster_name": "datadog_trace_collector",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 8126
}
}
}
}
]
}
]
}
}
EOF
}
}
}
```

deploy a connect enabled service with similar tracing configuration and perform HTTP requests to initiate tracing.

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 5
services = 3
build:
prerelease =
revision = 3111cb8c
version = 1.8.0
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 2
goroutines = 24350
max_procs = 2
os = linux
version = go1.14.4
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 49
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 3175
members = 34
query_queue = 0
query_time = 1

```

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 3111cb8c
version = 1.8.0
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.101.3.188:8300
server = true
raft:
applied_index = 708336420
commit_index = 708336420
fsm_pending = 0
last_contact = 57.987616ms
last_log_index = 708336420
last_log_term = 95
last_snapshot_index = 708323646
last_snapshot_term = 95
latest_configuration = [{Suffrage:Voter ID:04b87f04-ce07-0976-b4de-b29a3613b21a Address:10.101.4.10:8300} {Suffrage:Voter ID:daa06ae3-ea15-6b9e-791c-da38a2b66572 Address:10.101.3.188:8300} {Suffrage:Voter ID:60d8cc9c-b651-5e4f-1425-cfce296456e9 Address:10.101.4.50:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 95
runtime:
arch = amd64
cpu_count = 2
goroutines = 6536
max_procs = 2
os = linux
version = go1.14.4
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 49
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 3175
members = 34
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 211
members = 3
query_queue = 0
query_time = 1

```

### Operating system and Environment details

AmazonLinux 2

### Log Fragments

None

-----

This problem seems to be at https://github.com/hashicorp/consul/blob/v1.8.0/agent/xds/listeners.go#L933 which is suppressing traces at listener level. I tried to build a binary without this particular `RandomSampling` configuration and tracing started working again.

It would be helpful to control sampling using configuration options. We like to initiate tracing with every request that lands on the internet facing listener and then let the destination service decide if tracing should continue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.