RandomSampling on ingress gateway is suppressing distributed tracing
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
Ingress Gateway listener requires use of Envoy's `x-client-trace-id` header to initiate the trace. Without this header the requests are not traced at all.
#### Reproduction Steps
Create a new ingress gateway with tracing configuration. This example uses datadog tracer:
Ingress Gateway Service Config
```
service {
name = "igw"
port = 9999
kind = "ingress-gateway"
proxy {
config {
envoy_dogstatsd_url = "udp://127.0.0.1:8125"
envoy_tracing_json = <<-EOF
{
"http": {
"name": "envoy.tracers.datadog",
"config": {
"collector_cluster": "datadog_trace_collector",
"service_name": "igw"
}
}
}
EOF
envoy_extra_static_clusters_json = <<-EOF
{
"name": "datadog_trace_collector",
"type": "STATIC",
"connect_timeout": "1s",
"upstream_connection_options": {
"tcp_keepalive": {}
},
"load_assignment": {
"cluster_name": "datadog_trace_collector",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 8126
}
}
}
}
]
}
]
}
}
EOF
}
}
}
```
deploy a connect enabled service with similar tracing configuration and perform HTTP requests to initiate tracing.
### Consul info for both Client and Server
Client info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 5
services = 3
build:
prerelease =
revision = 3111cb8c
version = 1.8.0
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 2
goroutines = 24350
max_procs = 2
os = linux
version = go1.14.4
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 49
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 3175
members = 34
query_queue = 0
query_time = 1
```
Server info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 3111cb8c
version = 1.8.0
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.101.3.188:8300
server = true
raft:
applied_index = 708336420
commit_index = 708336420
fsm_pending = 0
last_contact = 57.987616ms
last_log_index = 708336420
last_log_term = 95
last_snapshot_index = 708323646
last_snapshot_term = 95
latest_configuration = [{Suffrage:Voter ID:04b87f04-ce07-0976-b4de-b29a3613b21a Address:10.101.4.10:8300} {Suffrage:Voter ID:daa06ae3-ea15-6b9e-791c-da38a2b66572 Address:10.101.3.188:8300} {Suffrage:Voter ID:60d8cc9c-b651-5e4f-1425-cfce296456e9 Address:10.101.4.50:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 95
runtime:
arch = amd64
cpu_count = 2
goroutines = 6536
max_procs = 2
os = linux
version = go1.14.4
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 49
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 3175
members = 34
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 211
members = 3
query_queue = 0
query_time = 1
```
### Operating system and Environment details
AmazonLinux 2
### Log Fragments
None
-----
This problem seems to be at https://github.com/hashicorp/consul/blob/v1.8.0/agent/xds/listeners.go#L933 which is suppressing traces at listener level. I tried to build a binary without this particular `RandomSampling` configuration and tracing started working again.
It would be helpful to control sampling using configuration options. We like to initiate tracing with every request that lands on the internet facing listener and then let the destination service decide if tracing should continue.
Contributor guide
Assessment
This issue has not been assessed yet.