firecracker-microvm / firecracker-microvm/firecracker-containerd

Failed to extract layer from example remote snapshotter

Open
#709 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.9k
Forks
247
PR merge metrics
No merged PRs in 30d

Description

Hi! I'm David and I am interested in making use of the `firecracker-containerd` in the context of remote snapshots using the `stargz` snapshotter.

I have followed the setup provided for running containers using remote snapshots as described below, but came across an issue when creating the task for the stargz based image. Further details are provided below and I am most thankful for your time!

# Setup details
## System
```bash
$ uname -a
Linux node-1.dbuzatu-138770.faas-sched-pg0.cloudlab.umass.edu 5.4.0-100-generic
#113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
```
```bash
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
```
## Installation
Installed `main` version of `firecracker-containerd` by following the steps from [quickstart](https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/getting-started.md).
- Docker has been installed by following [repository installation](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) steps.
- Golang has been installed following GoLang's installation page for [Linux](https://go.dev/doc/install).

Using firecracker-containerd to start a Docker based container using the following script works as expected:
```bash
sudo firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
run \
--snapshotter devmapper \
--runtime aws.firecracker \
--rm --tty --net-host \
docker.io/library/busybox:latest busybox-test
```

# Issue (UPDATED)
After following the [remote snapshotter](https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/remote-snapshotter-getting-started.md) setup guide, running the command:
```bash
./remote-snapshotter ghcr.io/firecracker-microvm/firecracker-containerd/amazonlinux:latest-esgz
```
results in the following output:
```bash
Creating VM
Setting docker credential metadata
Pulling the image
Creating a container
Creating a task
failed to create shim task: failed to create task: rpc error: code = Unknown desc = OCI runtime create failed: unable to retrieve OCI runtime error (open /container/remoteSnapshotterDemo/log.json: no such file or directory): runc did not terminate successfully: exit status 2: unknown
```

# Firecracker-containerd logs [gist](https://gist.github.com/DavidBuzatu-Marian/b9ae479a6d605ec42f4b03ea0800f998):
```bash
time="2022-11-11T16:42:32.581939242-05:00" level=debug msg="create VM request: VMID:\"vm1\" "
time="2022-11-11T16:42:32.582002348-05:00" level=debug msg="using namespace: vm1"
time="2022-11-11T16:42:32.582389912-05:00" level=info msg="successfully started shim (git commit: 24f1fcf99ebf6edcb94edd71a2affbcdae6b08e7)." runtime=aws.firecracker task_id=remoteSnapshotterDemo vmID=vm1
time="2022-11-11T16:42:32.584471369-05:00" level=debug msg="creating task" bundle=/run/firecracker-containerd/io.containerd.runtime.v2.task/vm1/remoteSnapshotterDemo checkpoint= runtime=aws.firecracker stderr=/run/containerd/fifo/2512387503/remoteSnapshotterDemo-stderr stdin=/run/containerd/fifo/2512387503/remoteSnapshotterDemo-stdin stdout=/run/containerd/fifo/2512387503/remoteSnapshotterDemo-stdout task_id=remoteSnapshotterDemo terminal=false vmID=vm1
time="2022-11-11T16:42:32.584717937-05:00" level=debug msg="noop operation returning shim dir for JailPath" jailer=noop runtime=aws.firecracker vmID=vm1
time="2022-11-11T16:42:32.642205537-05:00" level=debug msg="[ 5.161394] agent[796]: time=\"2022-11-11T21:42:32Z\" level=info msg=create ExecID= TaskID=remoteSnapshotterDemo" jailer=noop runtime=aws.firecracker vmID=vm1 vmm_stream=stdout
time="2022-11-11T16:42:32.686956668-05:00" level=debug ExecID= TaskID=remoteSnapshotterDemo attempt=1 direction=read error="temporary vsock dial failure: vsock ack message failure: failed to read \"OK \" within 1s: EOF" runtime=aws.firecracker stream=stdout vmID=vm1
time="2022-11-11T16:42:32.687087921-05:00" level=debug ExecID= TaskID=remoteSnapshotterDemo attempt=1 direction=write error="temporary vsock dial failure: vsock ack message failure: failed to read \"OK \" within 1s: EOF" runtime=aws.firecracker stream=stdin vmID=vm1
time="2022-11-11T16:42:32.687120563-05:00" level=debug ExecID= TaskID=remoteSnapshotterDemo attempt=1 direction=read error="temporary vsock dial failure: vsock ack message failure: failed to read \"OK \" within 1s: EOF" runtime=aws.firecracker stream=stderr vmID=vm1
time="2022-11-11T16:42:32.694295404-05:00" level=debug msg="[ 5.213658] systemd[1]: tmp-containerd\\x2dmount4037457222.mount: Succeeded." jailer=noop runtime=aws.firecracker vmID=vm1 vmm_stream=stdout
time="2022-11-11T16:42:32.786220731-05:00" level=debug msg="begin copying io" ExecID= TaskID=remoteSnapshotterDemo runtime=aws.firecracker stream=stdout vmID=vm1
time="2022-11-11T16:42:32.786710465-05:00" level=debug msg="begin copying io" ExecID= TaskID=remoteSnapshotterDemo runtime=aws.firecracker stream=stdin vmID=vm1
time="2022-11-11T16:42:32.786698280-05:00" level=debug msg="begin copying io" ExecID= TaskID=remoteSnapshotterDemo runtime=aws.firecracker stream=stderr vmID=vm1
time="2022-11-11T16:42:32.787360053-05:00" level=debug msg="[ 5.306653] systemd[1]: container-remoteSnapshotterDemo-rootfs.mount: Succeeded." jailer=noop runtime=aws.firecracker vmID=vm1 vmm_stream=stdout
time="2022-11-11T16:42:32.793437365-05:00" level=error error="failed to create task: rpc error: code = Unknown desc = OCI runtime create failed: unable to retrieve OCI runtime error (open /container/remoteSnapshotterDemo/log.json: no such file or directory): runc did not terminate successfully: exit status 2" runtime=aws.firecracker task_id=remoteSnapshotterDemo vmID=vm1
time="2022-11-11T16:42:32.793522671-05:00" level=debug msg="copied 0" ExecID= TaskID=remoteSnapshotterDemo runtime=aws.firecracker stream=stdin vmID=vm1
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.