liquidmetal-dev / liquidmetal-dev/flintlock
Flaky e2e test
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 73
- Avg merge
- 17h 24m
- Merged PRs (30d)
- 30
Description
**What happened:**
E2E test is not consistent, as I see because it hits rate limit when it tries to pull the image.
**What did you expect to happen:**
E2E test is consistent. Potential fix: if it hits rate limit, wait 2 minutes (or I don't know long, check service rate limiting docs). This extra would occur only if it hits rate limit, so it shouldn't cause significant slow down on average.
**How to reproduce it:**

**Anything else you would like to add:**
https://github.com/weaveworks-liquidmetal/flintlock/actions/runs/3745123487/jobs/6359213483
```
time="2022-12-21T00:04:03Z" level=debug msg="do request" digest="sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.6.14+unknown request.method=GET size=529 url="https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6"
time="2022-12-21T00:04:03Z" level=debug msg="fetch response received" digest="sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.cache-control=no-cache response.header.content-type=application/json response.status="429 Too Many Requests" size=529 url="https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6"
time="2022-12-21T00:04:03Z" level=debug msg="do request" digest="sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.6.14+unknown request.method=GET size=529 url="https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6"
time="2022-12-21T00:04:03Z" level=debug msg="fetch response received" digest="sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.cache-control=no-cache response.header.content-type=application/json response.status="429 Too Many Requests" size=529 url="https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6"
time="2022-12-21T00:04:03Z" level=error msg="failed executing plan" controller=microvm execution_id=01GMRZX0ZVFSBF1TZYFT689F28 execution_time=2.244781908s num_steps=2 plan_name=microvm_create_update
time="2022-12-21T00:04:03Z" level=info msg="[1/10] reconciliation failed, rescheduled for next attempt at 2022-12-21 00:04:23 +0000 UTC" action=reconcile controller=microvm vmid=ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B
time="2022-12-21T00:04:03Z" level=debug msg="saving microvm spec ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B" controller=microvm repo=containerd_microvm
time="2022-12-21T00:04:03.909277766Z" level=debug msg="(*service).Write started" ref=works.weave.flintlockd/microvm/ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B
time="2022-12-21T00:04:03Z" level=error msg="failed to reconcile vmid ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B: executing plan: executing plan steps: executing steps: executing step runtime_volume_mount: mount images docker.io/richardcase/ubuntu-bionic-test:cloudimage_v0.0.1 for volume use: getting image docker.io/richardcase/ubuntu-bionic-test:cloudimage_v0.0.1 for owner ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B: pulling image using containerd: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" controller=microvm
time="2022-12-21T00:04:03Z" level=debug msg="Getting spec for ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B" action=reconcile controller=microvm
time="2022-12-21T00:04:03Z" level=info msg="Starting reconciliation" action=reconcile controller=microvm vmid=ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B
time="2022-12-21T00:04:03Z" level=info msg="Generate plan" action=reconcile controller=microvm stage=plan vmid=ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B
time="2022-12-21T00:04:23.930717502Z" level=debug msg="event published" ns=flintlock topic=/microvm type=microvm.services.api.events.microvmspecupdated
time="2022-12-21T00:04:23Z" level=error msg="handling events: getting vmid from event data: name is required" controller=microvm
e2e_test.go:67:
Timed out after 120.000s.
Error: Assertion in callback at /root/work/flintlock/test/e2e/e2e_test.go:55 failed:
Expected
: /var/lib/flintlock/vm/ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B/firecracker.pid
to exist
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with test/e2e/e2e_test.go, especially the setup around lines 55-67, and review the linked GitHub Actions run. Confirm that Docker Hub's 429 response causes the timeout, then trace how the e2e test obtains the image. Done means the test remains reliable when image pulls hit the documented rate limit and still verifies the microVM startup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions, go
- Domain
- ci-cd, infrastructure, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100