`ConcreteService` failed to Start on burst
- Dominant language
- Go
- Stars
- 12
- Forks
- 1
- Avg merge
- 23m
- Merged PRs (30d)
- 1
Description
This issue is derived from https://github.com/grafana/mimir/issues/13287. I tried to add verbose logging in grafana/e2e to understand the current test's behavior, and interesting I found out following logs:
```go
service.go:137: docker port output for container e2e-mimir-test-mimir-6 port 8000 : Error: No public port '8000' published for e2e-mimir-test-mimir-6
service.go:148: docker ps output for container e2e-mimir-test-mimir-6 :
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
64a0d0c23fdd grafana/mimir:latest "mimir -blocks-stora…" 1 second ago Created e2e-mimir-test-mimir-5
3747c97179e6 grafana/mimir:latest "mimir -blocks-stora…" 1 second ago Created e2e-mimir-test-mimir-7
1e5ea6e7e6cd grafana/mimir:latest "mimir -memberlist.t…" 1 second ago Created e2e-mimir-test-mimir-18
5473a9d32ede grafana/mimir:latest "mimir -ingester.rin…" 1 second ago Created e2e-mimir-test-mimir-12
915dbe191526 grafana/mimir:latest "mimir -querier.fron…" 1 second ago Created e2e-mimir-test-mimir-2
948cd61cab73 grafana/mimir:latest "mimir -target=all -…" 1 second ago Created e2e-mimir-test-mimir-3
5541d9f11ce5 grafana/mimir:latest "mimir -blocks-stora…" 1 second ago Created e2e-mimir-test-mimir-8
0ea91f35a0e2 grafana/mimir:latest "mimir -blocks-stora…" 1 second ago Created e2e-mimir-test-mimir-4
5c37e25c95ba grafana/mimir:latest "mimir -blocks-stora…" 1 second ago Created e2e-mimir-test-mimir-14
07313fa3b322 grafana/mimir:latest "mimir -ingester.nat…" 1 second ago Created e2e-mimir-test-mimir-20
04e88bb734b5 grafana/mimir:latest "mimir -querier.fron…" 1 second ago Created e2e-mimir-test-mimir-16
0f506a94af00 grafana/mimir:latest "mimir -ingester.rin…" 1 second ago Created e2e-mimir-test-mimir-15
c199f1919e6a grafana/mimir:latest "mimir -querier.fron…" 1 second ago Up Less than a second 0.0.0.0:40225->8000/tcp, 0.0.0.0:35063->8080/tcp e2e-mimir-test-mimir-9
f4763efd17f7 grafana/mimir:latest "mimir -log.level=wa…" 1 second ago Created e2e-mimir-test-mimir-19
030af224cd47 grafana/mimir:latest "mimir -blocks-stora…" 1 second ago Up Less than a second 0.0.0.0:41455->8000/tcp, 0.0.0.0:33693->8080/tcp e2e-mimir-test-mimir-11
82320ec6ff2c grafana/mimir:latest "mimir -blocks-stora…" 1 second ago Up Less than a second 0.0.0.0:34847->8000/tcp, 0.0.0.0:33779->8080/tcp e2e-mimir-test-mimir-17
92e6eccdae75 grafana/mimir:latest "mimir -target=all -…" 1 second ago Up Less than a second 0.0.0.0:33913->8000/tcp, 0.0.0.0:45175->8080/tcp e2e-mimir-test-mimir-13
8be32e0aa44a grafana/mimir:latest "mimir -memberlist.p…" 1 second ago Up Less than a second 0.0.0.0:33387->8000/tcp, 0.0.0.0:40375->8080/tcp e2e-mimir-test-mimir-10
6c34d512de3d grafana/mimir:latest "mimir -ingester.rin…" 1 second ago Up Less than a second 0.0.0.0:38349->8000/tcp, 0.0.0.0:46357->8080/tcp e2e-mimir-test-mimir-6
0857fd4fcf8e grafana/mimir:latest "mimir -memberlist.t…" 3 seconds ago Up 2 seconds 0.0.0.0:34783->8000/tcp, 0.0.0.0:39147->8080/tcp e2e-mimir-test-mimir-1
c64b6a618260 minio/minio:RELEASE.2021-10-13T00-23-17Z "sh -c 'mkdir -p /da…" 4 seconds ago Up 3 seconds 0.0.0.0:44333->9000/tcp e2e-mimir-test-minio-9000
```
If we run `docker ps` after observing a `docker port` failure, we see the ports are actually assigned as intended. To address this out-of-sync issue, maybe we can implement a retry to mitigate the flakiness from the docker side?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in service.go around the docker port failure at line 137 and the diagnostic docker ps output around line 148. Reproduce the burst startup behavior, then determine how the port lookup should handle the observed timing gap; done means ConcreteService starts reliably when Docker assigns ports asynchronously.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- infrastructure, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100