Job with gateway not mounting secrets properly on envoy sidecar.
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Output from `nomad version`
`Nomad v1.0.4 (9294f35f9aa8dbb4acb6e85fa88e3e2534a3e41a)`
### Operating system and Environment details
Debian10, on-premise.
Output of `uname -a`
`Linux 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64 GNU/Linux`
### Issue
When running a connect enabled service, the envoy sidecar is unable to start because it cant detect the file `envoy_bootstrap.json` at the `secrets` path. However when using `nomad alloc fs` or the nomad web-ui I can clearly see that the file `secrets/envoy_bottstrap.json` exists.
If in the nomad client running the allocation I go to the `nomad/data/alloc///secrets` the directory is indeed empty. I've tried to "trick" the entrypoint of the envoy container to list the files under the `/secrets` path and as mention before it is empty. Using `docker inspect` I can see that the bind-mounts are properly working.
It may be relevant to note that mounted directories works properly on other containers, It just fails for connect enabled services.
### Reproduction steps
Run the job bellow.
### Job file (if appropriate)
Example from https://www.nomadproject.io/docs/job-specification/gateway#ingress-gateway
```hcl
job "ingress-demo" {
datacenters = ["dc1"]
group "ingress-group" {
network {
mode = "bridge"
port "inbound" {
static = 8080
to = 8080
}
}
service {
name = "my-ingress-service"
port = "8080"
connect {
gateway {
proxy {
}
ingress {
listener {
port = 8080
protocol = "tcp"
service {
name = "uuid-api"
}
}
}
}
}
}
}
group "generator" {
network {
mode = "host"
port "api" {}
}
service {
name = "uuid-api"
port = "api"
connect {
native = true
}
}
task "generate" {
driver = "docker"
config {
image = "hashicorpnomad/uuid-api:v5"
network_mode = "host"
}
env {
BIND = "0.0.0.0"
PORT = "${NOMAD_PORT_api}"
}
}
}
}
```
### Logs from connect-ingress-my-ingress-service container
```
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:305] initializing epoch 0 (base id=0, hot restart version=11.104)
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:307] statically linked extensions:
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:309] envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:309] envoy.thrift_proxy.filters: envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:309] envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:309] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:309] envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:309] envoy.stats_sinks: envoy.dog_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.statsd
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:309] envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:309] envoy.udp_listeners: quiche_quic_listener, raw_udp_listener
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:309] envoy.health_checkers: envoy.health_checkers.redis
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:309] envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.tcp_grpc, envoy.file_access_log, envoy.http_grpc_access_log, envoy.tcp_grpc_access_log
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:309] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:309] envoy.compression.compressor: envoy.compression.gzip.compressor
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:309] envoy.upstreams: envoy.filters.connection_pools.http.generic, envoy.filters.connection_pools.http.http, envoy.filters.connection_pools.http.tcp
[2021-03-03 08:43:11.890][9][info][main] [source/server/server.cc:309] envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.http.cache: envoy.extensions.http.cache.simple
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.dubbo_proxy.protocols: dubbo
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.compression.decompressor: envoy.compression.gzip.decompressor
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.dubbo_proxy.serializers: dubbo.hessian2
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.udp_packet_writers: udp_default_writer, udp_gso_batch_writer
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.dynamo, envoy.filters.http.ext_authz, envoy.filters.http.fault, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.gzip, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.oauth, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.squash, envoy.filters.http.tap, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.mysql_proxy, envoy.filters.network.postgres_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.rocketmq_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.quic_client_codec: quiche
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, tls
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.resolvers: envoy.ip
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.dubbo_proxy.route_matchers: default
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.bootstrap: envoy.extensions.network.socket_interface.default_socket_interface
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.quic_server_codec: quiche
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.thrift_proxy.transports: auto, framed, header, unframed
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:309] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2021-03-03 08:43:11.891][9][debug][misc] [source/common/filesystem/posix/filesystem_impl.cc:149] Unable to determine canonical path for /secrets/envoy_bootstrap.json: No such file or directory
[2021-03-03 08:43:11.891][9][critical][main] [source/server/server.cc:102] error initializing configuration '/secrets/envoy_bootstrap.json': Invalid path: /secrets/envoy_bootstrap.json
[2021-03-03 08:43:11.891][9][info][main] [source/server/server.cc:731] exiting
[2021-03-03 08:43:11.891][9][debug][main] [source/common/access_log/access_log_manager_impl.cc:19] destroyed access loggers
[2021-03-03 08:43:11.891][9][debug][init] [source/common/init/watcher_impl.cc:31] init manager Server destroyed
Invalid path: /secrets/envoy_bootstrap.json
```
### Nomad alloc fs
```bash
nomad alloc fs 1bca9a44 connect-ingress-my-ingress-service/secrets
Mode Size Modified Time Name
-rw-r--r-- 0 B 2021-03-03T09:42:35+01:00 .nomad-mount
-rw-r--r-- 7.5 KiB 2021-03-03T09:42:36+01:00 envoy_bootstrap.json
-r--r----- 36 B 2021-03-03T09:42:36+01:00 si_token
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the issue with the provided ingress gateway job, then compare `nomad alloc fs` output with the Docker bind mounts and the sidecar's `/secrets` path. The fix is complete when the Envoy sidecar can read `secrets/envoy_bootstrap.json` and start successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, linux
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100