liquidmetal-dev / liquidmetal-dev/flintlock

Flaky e2e test

Open
#611 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/testing kind/bug lifecycle/stale
Dominant language
Go
Stars
1.5k
Forks
73
Avg merge
17h 24m
Merged PRs (30d)
30

Description

**What happened:**
E2E test is not consistent, as I see because it hits rate limit when it tries to pull the image.

**What did you expect to happen:**
E2E test is consistent. Potential fix: if it hits rate limit, wait 2 minutes (or I don't know long, check service rate limiting docs). This extra would occur only if it hits rate limit, so it shouldn't cause significant slow down on average.

**How to reproduce it:**
![image](https://user-images.githubusercontent.com/183191/209303902-87db2ee7-fff2-4dfb-8c45-4349f395923b.png)

**Anything else you would like to add:**

https://github.com/weaveworks-liquidmetal/flintlock/actions/runs/3745123487/jobs/6359213483
```
time="2022-12-21T00:04:03Z" level=debug msg="do request" digest="sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.6.14+unknown request.method=GET size=529 url="https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6"
time="2022-12-21T00:04:03Z" level=debug msg="fetch response received" digest="sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.cache-control=no-cache response.header.content-type=application/json response.status="429 Too Many Requests" size=529 url="https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6"
time="2022-12-21T00:04:03Z" level=debug msg="do request" digest="sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.6.14+unknown request.method=GET size=529 url="https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6"
time="2022-12-21T00:04:03Z" level=debug msg="fetch response received" digest="sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.cache-control=no-cache response.header.content-type=application/json response.status="429 Too Many Requests" size=529 url="https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6"
time="2022-12-21T00:04:03Z" level=error msg="failed executing plan" controller=microvm execution_id=01GMRZX0ZVFSBF1TZYFT689F28 execution_time=2.244781908s num_steps=2 plan_name=microvm_create_update
time="2022-12-21T00:04:03Z" level=info msg="[1/10] reconciliation failed, rescheduled for next attempt at 2022-12-21 00:04:23 +0000 UTC" action=reconcile controller=microvm vmid=ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B
time="2022-12-21T00:04:03Z" level=debug msg="saving microvm spec ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B" controller=microvm repo=containerd_microvm
time="2022-12-21T00:04:03.909277766Z" level=debug msg="(*service).Write started" ref=works.weave.flintlockd/microvm/ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B
time="2022-12-21T00:04:03Z" level=error msg="failed to reconcile vmid ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B: executing plan: executing plan steps: executing steps: executing step runtime_volume_mount: mount images docker.io/richardcase/ubuntu-bionic-test:cloudimage_v0.0.1 for volume use: getting image docker.io/richardcase/ubuntu-bionic-test:cloudimage_v0.0.1 for owner ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B: pulling image using containerd: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" controller=microvm
time="2022-12-21T00:04:03Z" level=debug msg="Getting spec for ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B" action=reconcile controller=microvm
time="2022-12-21T00:04:03Z" level=info msg="Starting reconciliation" action=reconcile controller=microvm vmid=ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B
time="2022-12-21T00:04:03Z" level=info msg="Generate plan" action=reconcile controller=microvm stage=plan vmid=ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B
time="2022-12-21T00:04:23.930717502Z" level=debug msg="event published" ns=flintlock topic=/microvm type=microvm.services.api.events.microvmspecupdated
time="2022-12-21T00:04:23Z" level=error msg="handling events: getting vmid from event data: name is required" controller=microvm
e2e_test.go:67:
Timed out after 120.000s.
Error: Assertion in callback at /root/work/flintlock/test/e2e/e2e_test.go:55 failed:
Expected
: /var/lib/flintlock/vm/ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B/firecracker.pid
to exist
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with test/e2e/e2e_test.go, especially the setup around lines 55-67, and review the linked GitHub Actions run. Confirm that Docker Hub's 429 response causes the timeout, then trace how the e2e test obtains the image. Done means the test remains reliable when image pulls hit the documented rate limit and still verifies the microVM startup.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, go
Domain
ci-cd, infrastructure, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.