grafana / grafana/e2e

`ConcreteService` failed to Start on burst

Open
#57 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
12
Forks
1
Avg merge
23m
Merged PRs (30d)
1

Description

This issue is derived from https://github.com/grafana/mimir/issues/13287. I tried to add verbose logging in grafana/e2e to understand the current test's behavior, and interesting I found out following logs:

```go
service.go:137: docker port output for container e2e-mimir-test-mimir-6 port 8000 : Error: No public port '8000' published for e2e-mimir-test-mimir-6
service.go:148: docker ps output for container e2e-mimir-test-mimir-6 :
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
64a0d0c23fdd grafana/mimir:latest "mimir -blocks-stora…" 1 second ago Created e2e-mimir-test-mimir-5
3747c97179e6 grafana/mimir:latest "mimir -blocks-stora…" 1 second ago Created e2e-mimir-test-mimir-7
1e5ea6e7e6cd grafana/mimir:latest "mimir -memberlist.t…" 1 second ago Created e2e-mimir-test-mimir-18
5473a9d32ede grafana/mimir:latest "mimir -ingester.rin…" 1 second ago Created e2e-mimir-test-mimir-12
915dbe191526 grafana/mimir:latest "mimir -querier.fron…" 1 second ago Created e2e-mimir-test-mimir-2
948cd61cab73 grafana/mimir:latest "mimir -target=all -…" 1 second ago Created e2e-mimir-test-mimir-3
5541d9f11ce5 grafana/mimir:latest "mimir -blocks-stora…" 1 second ago Created e2e-mimir-test-mimir-8
0ea91f35a0e2 grafana/mimir:latest "mimir -blocks-stora…" 1 second ago Created e2e-mimir-test-mimir-4
5c37e25c95ba grafana/mimir:latest "mimir -blocks-stora…" 1 second ago Created e2e-mimir-test-mimir-14
07313fa3b322 grafana/mimir:latest "mimir -ingester.nat…" 1 second ago Created e2e-mimir-test-mimir-20
04e88bb734b5 grafana/mimir:latest "mimir -querier.fron…" 1 second ago Created e2e-mimir-test-mimir-16
0f506a94af00 grafana/mimir:latest "mimir -ingester.rin…" 1 second ago Created e2e-mimir-test-mimir-15
c199f1919e6a grafana/mimir:latest "mimir -querier.fron…" 1 second ago Up Less than a second 0.0.0.0:40225->8000/tcp, 0.0.0.0:35063->8080/tcp e2e-mimir-test-mimir-9
f4763efd17f7 grafana/mimir:latest "mimir -log.level=wa…" 1 second ago Created e2e-mimir-test-mimir-19
030af224cd47 grafana/mimir:latest "mimir -blocks-stora…" 1 second ago Up Less than a second 0.0.0.0:41455->8000/tcp, 0.0.0.0:33693->8080/tcp e2e-mimir-test-mimir-11
82320ec6ff2c grafana/mimir:latest "mimir -blocks-stora…" 1 second ago Up Less than a second 0.0.0.0:34847->8000/tcp, 0.0.0.0:33779->8080/tcp e2e-mimir-test-mimir-17
92e6eccdae75 grafana/mimir:latest "mimir -target=all -…" 1 second ago Up Less than a second 0.0.0.0:33913->8000/tcp, 0.0.0.0:45175->8080/tcp e2e-mimir-test-mimir-13
8be32e0aa44a grafana/mimir:latest "mimir -memberlist.p…" 1 second ago Up Less than a second 0.0.0.0:33387->8000/tcp, 0.0.0.0:40375->8080/tcp e2e-mimir-test-mimir-10
6c34d512de3d grafana/mimir:latest "mimir -ingester.rin…" 1 second ago Up Less than a second 0.0.0.0:38349->8000/tcp, 0.0.0.0:46357->8080/tcp e2e-mimir-test-mimir-6
0857fd4fcf8e grafana/mimir:latest "mimir -memberlist.t…" 3 seconds ago Up 2 seconds 0.0.0.0:34783->8000/tcp, 0.0.0.0:39147->8080/tcp e2e-mimir-test-mimir-1
c64b6a618260 minio/minio:RELEASE.2021-10-13T00-23-17Z "sh -c 'mkdir -p /da…" 4 seconds ago Up 3 seconds 0.0.0.0:44333->9000/tcp e2e-mimir-test-minio-9000
```

If we run `docker ps` after observing a `docker port` failure, we see the ports are actually assigned as intended. To address this out-of-sync issue, maybe we can implement a retry to mitigate the flakiness from the docker side?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in service.go around the docker port failure at line 137 and the diagnostic docker ps output around line 148. Reproduce the burst startup behavior, then determine how the port lookup should handle the observed timing gap; done means ConcreteService starts reliably when Docker assigns ports asynchronously.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
infrastructure, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.