firecracker-microvm / firecracker-microvm/firecracker-containerd

[ARM platform]Remote snapshotter test failed on pulling image due to microVM time out of sync

Open
#737 2 comments 0 reactions 0 assignees View on GitHub
kind/bug
Dominant language
Go
Stars
2.9k
Forks
247
PR merge metrics
No merged PRs in 30d

Description

On ARM64 platform, remote snapshotter test failed on pulling image due to microVM time out of sync.
Failed CI build [link](https://buildkite.com/firecracker-microvm/firecracker-containerd-arm/builds/48#018757ab-fc33-40f0-aa99-d37422fd9bfd).
When CI tried to run test [TestSnapshotterMetrics_Isolated](https://github.com/firecracker-microvm/firecracker-containerd/blob/main/snapshotter/metrics_integ_test.go#L43) which will pull an image and unpack in remote snapshotter, it failed with below issue on pulling image(the image pulling fine on ARM host with ctr).
```
=== RUN TestSnapshotterMetrics_Isolated
metrics_integ_test.go:54:
Error Trace: metrics_integ_test.go:54
Error: Received unexpected error:
Failed to pull image on microVM[0]: failed to extract layer sha256:fdb3c0ecba2ee0b2b39f778f7da3beb4ee4c75f6f4b8a083211b12971fde4ad6: failed to mount /var/lib/containerd/tmpmounts/containerd-mount3440161598: no such file or directory: unknown
Test: TestSnapshotterMetrics_Isolated
```
Checked the [containerd.log](https://buildkite.com/organizations/firecracker-microvm/pipelines/firecracker-containerd-arm/builds/48/jobs/018757ab-fc33-40f0-aa99-d37422fd9bfd/artifacts/018757be-926c-450c-a4a7-f5a4ad9642c7) for the test, found that there is `Error x509: certificate has expired or is not yet valid: current time 2022-08-07T13:25:13Z is before 2023-02-21T00:00:00Z` seems the microVM time is not synced right.
```
{\"key\":\"0/1/extract-101125884-ByGC sha256:fdb3c0ecba2ee0b2b39f778f7da3beb4ee4c75f6f4b8a083211b12971fde4ad6\",\"level\":\"info\",\"mountpoint\":\"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/1/fs\",\"msg\":\"Received status code: 401 Unauthorized. Refreshing creds...\",\"parent\":\"\",\"src\":\"ghcr.io/firecracker-microvm/firecracker-containerd/amazonlinux:latest-esgz/sha256:efc8b66d208d6eaa2e24799081e13c035f25ad585cec5d478845a744f98324b8\",\"time\":\"2022-08-07T13:25:13.491954298Z\"}" jailer=noop runtime=aws.firecracker vmID=0 vmm_stream=stdout
time="2023-04-06T18:05:06.129064764Z" level=debug msg="[ 5.441207] containerd-stargz-grpc[744]: {\"error\":\"Get \\\"https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:efc8b66d208d6eaa2e24799081e13c035f25ad585cec5d478845a744f98324b8?se=2023-04-06T18%3A15%3A00Z\\u0026sig=8ccbSSbYFbIlb%2Fr9eh5ptZlKm5W1pkw50WfKqVLTY4A%3D\\u0026sp=r\\u0026spr=https\\u0026sr=b\\u0026sv=2019-12-12\\\": x509: certificate has expired or is not yet valid: current time 2022-08-07T13:25:13Z is before 2023-02-21T00:00:00Z\"
```
Tried to set the microVM time in the [commit](https://github.com/firecracker-microvm/firecracker-containerd/pull/736/commits/343044b95baa86fa405ecd7d68f04413531cea1e), but didn’t make effect, rerun still had old time.
Strange part is AMD [platform build ](https://buildkite.com/firecracker-microvm/firecracker-containerd-amd/builds/95#018757ab-ed50-407d-9a00-3eb15ce80c85)has the right time in MicroVM.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.