hashicorp / hashicorp/nomad

Nomad not GCing old connect sidecar proxy containers

Open
#10,503 0 comments 0 reactions 0 assignees View on GitHub
theme/consul/connect theme/driver/docker type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

While upgrading a VPS to v1.1.0-beta (from v1.0.4) yesterday I noticed a lot of old connect sidecar proxy images lingering around. These should get cleaned up like any other tasks. One thing to note is `sudo reboot` is the usual way nomad gets restarted on this machine.

```
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f253bd836433 envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 2 weeks ago Up 2 weeks connect-proxy-caddy-6cd4582e-f107-29da-fce7-e244a3616ca8
253ab9078177 gcr.io/google_containers/pause-amd64:3.1 "/pause" 2 weeks ago Up 2 weeks nomad_init_6cd4582e-f107-29da-fce7-e244a3616ca8
db1c325dd466 envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 2 weeks ago Up 2 weeks connect-proxy-ubs-32af591a-b24f-7742-1bdb-272577af55e5
dac2847f8381 gcr.io/google_containers/pause-amd64:3.1 "/pause" 2 weeks ago Up 2 weeks nomad_init_32af591a-b24f-7742-1bdb-272577af55e5
43cca8c2cf4e envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 2 weeks ago Up 2 weeks connect-proxy-hclfmt-f500e4fb-5054-4c03-899a-f8bb551d536d
57e528fdcc2a gcr.io/google_containers/pause-amd64:3.1 "/pause" 2 weeks ago Up 2 weeks nomad_init_f500e4fb-5054-4c03-899a-f8bb551d536d
0b0c9d6c05e4 envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 2 weeks ago Up 2 weeks connect-proxy-mod-redirect-b9231645-14ea-2ee4-66ac-da196a36faf6
7b4836b5f064 gcr.io/google_containers/pause-amd64:3.1 "/pause" 2 weeks ago Up 2 weeks nomad_init_b9231645-14ea-2ee4-66ac-da196a36faf6
00ebf65b1785 envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 3 weeks ago Exited (137) 2 weeks ago connect-proxy-caddy-e2c9a50d-027b-28d1-54f0-80f47c7cb922
98d04623cf23 envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 3 weeks ago Exited (137) 2 weeks ago connect-proxy-mod-redirect-bad304f8-28ff-e15e-d405-7ff25690a2b9
7a9b36ecdacf envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 3 weeks ago Exited (137) 2 weeks ago connect-proxy-hclfmt-18aca177-e1e1-9ae1-ed71-bace2801fa17
01243b11ae00 envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 3 weeks ago Exited (137) 2 weeks ago connect-proxy-ubs-ec6a3eb9-d636-7448-5ecf-7d53f1c19ac9
ff49aad9a75a envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 4 weeks ago Exited (137) 3 weeks ago connect-proxy-ubs-d06e403f-b2f8-3fe2-cade-5b3f904f968e
e6eedfdd3266 envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 4 weeks ago Exited (137) 3 weeks ago connect-proxy-caddy-c1a19c31-7835-6041-fb78-c01f48d0b09a
3a9ec48f078d envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 4 weeks ago Exited (137) 3 weeks ago connect-proxy-hclfmt-eae9fafb-9432-faec-4e7c-122e88b67c21
fa2d0c416feb envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 4 weeks ago Exited (137) 3 weeks ago connect-proxy-mod-redirect-5c70ae62-b77f-2496-dbdc-ba276b419832
fe3bcea74ab2 envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 5 weeks ago Exited (137) 4 weeks ago connect-proxy-hclfmt-b539bbf5-72ab-4bb1-6170-c3775bcfb84f
59cfa975192c envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 5 weeks ago Exited (137) 4 weeks ago connect-proxy-ubs-ddf3f60d-7352-717b-5370-a2d66e9d7037
b4e79a8501bc envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 5 weeks ago Exited (137) 4 weeks ago connect-proxy-caddy-a5c4494b-1234-ca26-c0b3-4bc28514af84
bb4d1b270388 envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 5 weeks ago Exited (137) 4 weeks ago connect-proxy-mod-redirect-96e66b8b-85ee-7748-bba8-547dc6489b40
21768eef2735 envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 2 months ago Exited (137) 5 weeks ago connect-proxy-hclfmt-86543d62-1500-29e3-9313-1f27207068d4
1d877a3b9369 envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 2 months ago Exited (137) 5 weeks ago connect-proxy-ubs-055eafb2-57fc-fbe9-26e0-a7b911dce734
34c0e46be1ca envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 2 months ago Exited (137) 5 weeks ago connect-proxy-mod-redirect-98ccef05-48cc-6483-44aa-5523a0853efb
a8fe5fdff3c2 envoyproxy/envoy:v1.16.0 "/docker-entrypoint.…" 2 months ago Exited (137) 5 weeks ago connect-proxy-caddy-7057d6f8-b3d7-e2f8-c4ae-9d58af4971c2
```

Contributor guide

No contributing guide indexed for this repository

Research direction

The report identifies Docker containers named connect-proxy-* and nomad_init_* but no source file or test. Start by tracing Nomad's Docker task cleanup path for Connect sidecars and verify behavior around a restart; done means exited old sidecar containers are removed while active proxies remain.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.