hashicorp / hashicorp/consul

Sidecar proxy with TLSV1_ALERT_PROTOCOL_VERSION error

Open
#17,362 11 comments 1 reaction 0 assignees View on GitHub
type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

### Nomad version
Output from `nomad version`
```
Nomad v1.5.3
BuildDate 2023-04-04T20:09:50Z
Revision 434f7a1745c6304d607562daa9a4a635def7153f
```

Output from `consul version`
```
Consul v1.15.2
Revision 5e08e229
Build Date 2023-03-30T17:51:19Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
```

### Operating system and Environment details

Ubuntu 22.04.2

### Issue

When attempting to use consul connect on a job the side car never reaches a healthy state and prints out every few seconds:
```
[2023-04-25 16:59:09.648][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 77s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
```

### Reproduction steps

Have consul connect enabled and configured, deploy a job which uses an envoy sidecar for consul connect and start the job. After an init period the sidecar will start printing out the warning messages every few seconds. Both Consul and Nomad have ACL enabled.

Consul TLS, Address, and Ports configs (These are templated from ansible):
```
addresses {
dns = "127.0.0.1 {{ cluster_internal_ip }}"
http = "127.0.0.1 {{ cluster_internal_ip }}"
https = "127.0.0.1 {{ cluster_internal_ip }}"
grpc = "127.0.0.1 {{ cluster_internal_ip }}"
grpc_tls = "127.0.0.1 {{ cluster_internal_ip }}"
}

ports {
dns = {{ consul_dns_port | int }}
http = {{ consul_http_port | int }}
https = {{ consul_https_port | int }}
grpc = {{ consul_grpc_port | int }}
grpc_tls = {{ consul_grpc_tls_port | int }}
}

tls {
defaults {
tls_min_version = "TLSv1_3"

ca_file = "{{ consul_tls_dir }}/certs/{{ cluster_base_domain }}-agent-ca.pem"
}

internal_rpc {
verify_server_hostname = true
verify_incoming = true
verify_outgoing = true
}

https {
tls_min_version = "TLSv1_3"
ca_file = "{{ consul_tls_dir }}/certs/ca.pem"
cert_file = "{{ consul_tls_dir }}/certs/local.pem"
key_file = "{{ consul_tls_dir }}/private/local-key.pem"
}

grpc {
use_auto_cert = true
verify_incoming = true
}
}
```

Consul server additonally has:
```
tls {
defaults {
cert_file = "{{ consul_tls_dir }}/certs/{{ cluster_datacenter }}-server-{{ cluster_base_domain }}.pem"
key_file = "{{ consul_tls_dir }}/private/{{ cluster_datacenter }}-server-{{ cluster_base_domain }}-key.pem"
}
}

auto_encrypt {
allow_tls = true
}

connect {
enabled = true
}
```

Nomad Consul config:
```
consul {
allow_unauthenticated = false

address = "127.0.0.1:{{ consul_https_port }}"
grpc_address = "127.0.0.1:{{ consul_grpc_tls_port }}"

ca_file = "{{ nomad_config_dir }}/.consul/ca.pem"
grpc_ca_file = "{{ nomad_config_dir }}/.consul/agent-ca.pem"

ssl = true
verify_ssl = true
share_ssl = true

auto_advertise = true
checks_use_advertise = true

server_service_name = "nomad"
client_service_name = "nomad-client"

server_auto_join = false
client_auto_join = false
}
```

#### Expected Result

Sidecar registers service with Consul and becomes healthy ready to serve traffic

#### Actual Result

Sidecar registers service with Consul and then does not come to a healthy state, Instead prints out warnings every few seconds like:

```
[2023-04-25 16:59:09.648][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 77s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
```

### Job file (if appropriate)

I'm currently attempting to run this job file:

[https://gitlab.com/carboncollins-cloud/monitoring/log-management/-/blob/main/job.template.nomad.hcl](https://gitlab.com/carboncollins-cloud/monitoring/log-management/-/blob/main/job.template.nomad.hcl)

Sidecar Logs in full:
```
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:406] statically linked extensions:
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] quic.http_server_connection: quic.http_server_connection.default
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.dubbo_proxy.serializers: dubbo.hessian2
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.route.early_data_policy: envoy.route.early_data_policy.default
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.formatter: envoy.formatter.metadata, envoy.formatter.req_without_query
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.header
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] network.connection.client: default, envoy_internal
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple
[2023-04-25 16:57:51.484][1][info][main] [source/server/server.cc:408] envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo
[2023-04-25 16:57:51.485][1][info][main] [source/server/server.cc:408] envoy.quic.connection_id_generator: envoy.quic.deterministic_connection_id_generator
[2023-04-25 16:57:51.485][1][info][main] [source/server/server.cc:408] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
[2023-04-25 16:57:51.485][1][info][main] [source/server/server.cc:408] envoy.tracers: envoy.dynamic.ot, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2023-04-25 16:57:51.485][1][info][main] [source/server/server.cc:408] envoy.matching.http.input: envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san
[2023-04-25 16:57:51.485][1][info][main] [source/server/server.cc:408] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2023-04-25 16:57:51.485][1][info][main] [source/server/server.cc:408] envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2023-04-25 16:57:51.485][1][info][main] [source/server/server.cc:408] envoy.dubbo_proxy.protocols: dubbo
[2023-04-25 16:57:51.485][1][info][main] [source/server/server.cc:408] envoy.health_checkers: envoy.health_checkers.redis, envoy.health_checkers.thrift
[2023-04-25 16:57:51.485][1][info][main] [source/server/server.cc:408] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router
[2023-04-25 16:57:51.486][1][info][main] [source/server/server.cc:408] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
[2023-04-25 16:57:51.486][1][info][main] [source/server/server.cc:408] envoy.common.key_value: envoy.key_value.file_based
[2023-04-25 16:57:51.486][1][info][main] [source/server/server.cc:408] envoy.resolvers: envoy.ip
[2023-04-25 16:57:51.486][1][info][main] [source/server/server.cc:408] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2023-04-25 16:57:51.486][1][info][main] [source/server/server.cc:408] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2023-04-25 16:57:51.486][1][info][main] [source/server/server.cc:408] envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
[2023-04-25 16:57:51.486][1][info][main] [source/server/server.cc:408] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san
[2023-04-25 16:57:51.486][1][info][main] [source/server/server.cc:408] envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso
[2023-04-25 16:57:51.487][1][info][main] [source/server/server.cc:408] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2023-04-25 16:57:51.487][1][info][main] [source/server/server.cc:408] envoy.upstreams: envoy.filters.connection_pools.tcp.generic
[2023-04-25 16:57:51.487][1][info][main] [source/server/server.cc:408] envoy.regex_engines: envoy.regex_engines.google_re2
[2023-04-25 16:57:51.487][1][info][main] [source/server/server.cc:408] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2023-04-25 16:57:51.487][1][info][main] [source/server/server.cc:408] envoy.listener_manager_impl: envoy.listener_manager_impl.default
[2023-04-25 16:57:51.487][1][info][main] [source/server/server.cc:408] envoy.connection_handler: envoy.connection_handler.default
[2023-04-25 16:57:51.487][1][info][main] [source/server/server.cc:408] envoy.path.match: envoy.path.match.uri_template.uri_template_matcher
[2023-04-25 16:57:51.487][1][info][main] [source/server/server.cc:408] envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter
[2023-04-25 16:57:51.487][1][info][main] [source/server/server.cc:408] envoy.load_balancing_policies: envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.random, envoy.load_balancing_policies.round_robin
[2023-04-25 16:57:51.488][1][info][main] [source/server/server.cc:408] envoy.filters.http.upstream: envoy.buffer, envoy.filters.http.admission_control, envoy.filters.http.buffer, envoy.filters.http.upstream_codec
[2023-04-25 16:57:51.488][1][info][main] [source/server/server.cc:408] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
[2023-04-25 16:57:51.488][1][info][main] [source/server/server.cc:408] envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2023-04-25 16:57:51.488][1][info][main] [source/server/server.cc:408] envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2023-04-25 16:57:51.488][1][info][main] [source/server/server.cc:408] envoy.http.header_validators: envoy.http.header_validators.envoy_default
[2023-04-25 16:57:51.488][1][info][main] [source/server/server.cc:408] envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2023-04-25 16:57:51.488][1][info][main] [source/server/server.cc:408] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2023-04-25 16:57:51.488][1][info][main] [source/server/server.cc:408] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
[2023-04-25 16:57:51.488][1][info][main] [source/server/server.cc:408] envoy.thrift_proxy.transports: auto, framed, header, unframed
[2023-04-25 16:57:51.489][1][info][main] [source/server/server.cc:408] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
[2023-04-25 16:57:51.489][1][info][main] [source/server/server.cc:408] envoy.matching.action: envoy.matching.actions.format_string, filter-chain-name
[2023-04-25 16:57:51.489][1][info][main] [source/server/server.cc:408] envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy
[2023-04-25 16:57:51.489][1][info][main] [source/server/server.cc:408] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
[2023-04-25 16:57:51.489][1][info][main] [source/server/server.cc:408] envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation
[2023-04-25 16:57:51.489][1][info][main] [source/server/server.cc:408] envoy.matching.input_matchers: envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip
[2023-04-25 16:57:51.489][1][info][main] [source/server/server.cc:408] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
[2023-04-25 16:57:51.489][1][info][main] [source/server/server.cc:408] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2023-04-25 16:57:51.489][1][info][main] [source/server/server.cc:408] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
[2023-04-25 16:57:51.489][1][info][main] [source/server/server.cc:408] envoy.request_id: envoy.request_id.uuid
[2023-04-25 16:57:51.489][1][info][main] [source/server/server.cc:408] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
[2023-04-25 16:57:51.489][1][info][main] [source/server/server.cc:408] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2023-04-25 16:57:51.499][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.http2_protocol_options' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2023-04-25 16:57:51.499][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.bootstrap.v3.Admin Using deprecated option 'envoy.config.bootstrap.v3.Admin.access_log_path' from file bootstrap.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2023-04-25 16:57:51.499][1][info][main] [source/server/server.cc:456] HTTP header map info:
[2023-04-25 16:57:51.506][1][info][main] [source/server/server.cc:459] request header map: 672 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-ot-span-context,x-request-id
[2023-04-25 16:57:51.506][1][info][main] [source/server/server.cc:459] request trailer map: 120 bytes:
[2023-04-25 16:57:51.506][1][info][main] [source/server/server.cc:459] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2023-04-25 16:57:51.506][1][info][main] [source/server/server.cc:459] response trailer map: 144 bytes: grpc-message,grpc-status
[2023-04-25 16:57:51.540][1][info][main] [source/server/server.cc:819] runtime: layers:
- name: base
static_layer:
re2.max_program_size.error_level: 1048576
[2023-04-25 16:57:51.543][1][info][admin] [source/server/admin/admin.cc:67] admin address: 127.0.0.2:19001
[2023-04-25 16:57:51.545][1][info][config] [source/server/configuration_impl.cc:131] loading tracing configuration
[2023-04-25 16:57:51.545][1][info][config] [source/server/configuration_impl.cc:91] loading 0 static secret(s)
[2023-04-25 16:57:51.545][1][info][config] [source/server/configuration_impl.cc:97] loading 1 cluster(s)
[2023-04-25 16:57:51.680][1][info][config] [source/server/configuration_impl.cc:101] loading 0 listener(s)
[2023-04-25 16:57:51.680][1][info][config] [source/server/configuration_impl.cc:113] loading stats configuration
[2023-04-25 16:57:51.681][1][info][runtime] [source/common/runtime/runtime_impl.cc:463] RTDS has finished initialization
[2023-04-25 16:57:51.681][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:222] cm init: initializing cds
[2023-04-25 16:57:51.682][1][warning][main] [source/server/server.cc:794] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
[2023-04-25 16:57:51.684][1][info][main] [source/server/server.cc:915] starting main dispatch loop
[2023-04-25 16:58:06.683][1][warning][config] [source/common/config/grpc_subscription_impl.cc:120] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.cluster.v3.Cluster
[2023-04-25 16:58:06.683][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:226] cm init: all clusters initialized
[2023-04-25 16:58:06.683][1][info][main] [source/server/server.cc:896] all clusters initialized. initializing init manager
[2023-04-25 16:58:21.688][1][warning][config] [source/common/config/grpc_subscription_impl.cc:120] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.listener.v3.Listener
[2023-04-25 16:58:21.688][1][info][config] [source/extensions/listener_managers/listener_manager/listener_manager_impl.cc:852] all dependencies initialized. starting workers
[2023-04-25 16:58:39.965][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 48s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 16:58:44.253][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 52s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 16:59:00.740][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 69s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 16:59:09.648][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 77s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 16:59:21.633][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 89s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 16:59:33.627][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 101s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 17:00:02.644][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 130s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 17:00:16.816][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 145s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 17:00:17.003][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 145s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 17:00:24.142][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 152s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 17:00:48.382][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 176s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 17:01:16.919][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 205s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 17:01:39.429][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 227s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 17:02:00.094][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 248s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 17:02:07.098][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 255s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 17:02:09.933][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 258s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 17:02:38.744][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 287s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2023-04-25 17:02:43.870][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 292s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION

```

### Nomad Server logs (if appropriate)

### Nomad Client logs (if appropriate)
I noticed that the client seems to be refusing the rpc connection but the port looks correct...

```
Apr 25 18:40:33 soc-se-07 nomad[1273]: 2023-04-25T18:40:33.849+0200 [INFO] agent: (runner) creating new runner (dry: false, once: false)
Apr 25 18:40:33 soc-se-07 nomad[1273]: 2023-04-25T18:40:33.851+0200 [INFO] agent: (runner) creating watcher
Apr 25 18:40:33 soc-se-07 nomad[1273]: 2023-04-25T18:40:33.852+0200 [INFO] agent: (runner) starting
Apr 25 18:40:33 soc-se-07 nomad[1273]: 2023-04-25T18:40:33.863+0200 [INFO] agent: (runner) rendered "(dynamic)" => "/opt/nomad/data/alloc/691d305c-6906-c5fe-5608-15591f9968b6/loki/loc>
Apr 25 18:40:34 soc-se-07 nomad[1273]: 2023-04-25T18:40:34.224+0200 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=9e54b97fcdb74055a3ffe5983f0e98b2384a43546a0ac4fe7760d56b2dda72d5
Apr 25 18:40:34 soc-se-07 nomad[1273]: 2023-04-25T18:40:34.560+0200 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=9e54b97fcdb74055a3ffe5983f0e98b2384a43546a0ac4fe7760d56b2dda72d5
Apr 25 18:40:34 soc-se-07 nomad[1273]: 2023-04-25T18:40:34.671+0200 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=691d305c-6906-c5fe-5608-15591f9968b6 task=loki type=Started msg="Task started by client" failed=false
Apr 25 18:54:35 soc-se-07 nomad[1273]: 2023-04-25T18:54:35.909+0200 [ERROR] client.alloc_runner.runner_hook: error connecting to grpc: alloc_id=691d305c-6906-c5fe-5608-15591f9968b6 error="dial tcp 127.0.0.1:8503: connect: connection refused" dest=127.0.0.1:8503
Apr 25 18:57:49 soc-se-07 nomad[1273]: 2023-04-25T18:57:49.978+0200 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=691d305c-6906-c5fe-5608-15591f9968b6 task=connect-proxy-loki type="Restart Signaled" msg="User requested running tasks to restart"
Apr 25 18:57:49 soc-se-07 nomad[1273]: 2023-04-25T18:57:49.978+0200 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=691d305c-6906-c5fe-5608-15591f9968b6 task=loki type="Restart Signaled" msg="User requested running tasks to restart" failed=false
Apr 25 18:57:50 soc-se-07 nomad[1273]: 2023-04-25T18:57:50.513+0200 [INFO] client.driver_mgr.docker: stopped container: container_id=9fc25f6992873cb32cfe7dc75a4173bea01082307830048
```

Contributor guide

Open the contributing guide

Research direction

Start with the Consul TLS and grpc configuration shown in the issue, then compare the sidecar's DeltaAggregatedResources warning with the configured grpc_tls address and port. Reproduce the job with the supplied Consul, Nomad, and Ubuntu versions; done means identifying the TLS protocol mismatch and documenting or implementing a verified fix so the sidecar becomes healthy.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
distributed-systems, networking, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.