containerd / containerd/accelerated-container-image

Overlaybd did not block when networking was not available for long time

Open
#214 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
470
Forks
87
Avg merge
2d 21h
Merged PRs (30d)
5

Description

### What happened in your environment?

We found a potential overlaybd bug that it returned incorrect data during networking was down. This could lead to application failures, in our case is Java failed to load class

### What did you expect to happen?

When networking is down, the class loading should be completely blocked until the network recovers. However, we currently see "Exception: java.lang.NoClassDefFoundError" and " error reading zip file" after retrying for 3+ minutes.

We suspect there's a bug in overlaybd that it returned some unexpected result but instead it should block until networking is recovered. given the following experiments we did:

1. We did `systemctl stop overlaybd-tcmu`, after which `jar` command would actually hang forever until overlaybd-tcmu recover
2. With a normal jar stored on a device-mapper block device, if we suspend the IO in the DM device, the jar command would hang forever until the IO suspension was removed

### How can we reproduce it?

* Step 1, build, convert and push a repro image using the following Dockerfile
```
FROM ubuntu:18.04
RUN apt-get update \
# TODO: upgrade to JAVA 11 in the next sprint
&& apt-get install -y openjdk-8-jdk git vim \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
RUN git clone https://github.com/macagua/example.java.helloworld.git && cd example.java.helloworld && javac HelloWorld/Main.java && jar cfme Main.jar Manifest.txt HelloWorld.Main HelloWorld/Main.class
RUN echo 'export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64' >> ~/.bashrc
```

* Step 2: rpull and bash into the container

```
/opt/overlaybd/snapshotter/ctr -n k8s.io rpull -u $USERNAME:$PASSWORD $IMAGE_REF

ctr -n k8s.io run --snapshotter=overlaybd --rm -t $IMAGE_REF test-jar bash

# In side the shell, run `jar` command to load the binary
```

* Step 3: shutdown the network, we did this by turning off the security group of the VM
* Step 4: inside the bash shell, run
```
jar vft ./example.java.helloworld/Main.jar

# After several minutes, we see "error reading zip file" error
root@ip-10-0-0-134:/# jar vft ./example.java.helloworld/Main.jar
/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar: error reading zip file
/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar: error reading zip file
Exception in thread "main"
Exception: java.lang.NoClassDefFoundError thrown from the UncaughtExceptionHandler in thread "main"

```

### What is the version of your Accelerated Container Image?

* overlaybd 0.6.10

### What is your OS environment?

ubuntu

### Are you willing to submit PRs to fix it?

- [ ] Yes, I am willing to fix it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.