aws / aws/eks-anywhere

cluster creation fails with "Failed to mount API filesystems, freezing"

Open
#838 6 comments 0 reactions 0 assignees View on GitHub
external priority/p1 triage/accepted
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:

```console
$ eksctl anywhere generate clusterconfig eks0 --provider docker --verbosity 10000 > eks0.yaml 2>eks0.log
$ eksctl anywhere create cluster -f eks0.yaml --verbosity 10000 2>&1| tee -a eks0.log
...
Error: failed to create cluster: error creating bootstrap cluster: error executing create cluster: exit status 1, try rerunning with --force-cleanup to force delete previously created bootstrap cluster
```
Only by looking separately at the `` container logs does one see the actual failure:

```console
$ docker logs $(docker ps -qf label=io.x-k8s.kind.role=control-plane)
INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v2
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from VM UUID.
INFO: faking /sys/class/dmi/id/product_name to be "kind"
INFO: faking /sys/class/dmi/id/product_uuid to be random
INFO: faking /sys/devices/virtual/dmi/id/product_uuid as well
INFO: setting iptables to detected mode: legacy
INFO: Detected IPv4 address: 172.18.0.2
INFO: Detected IPv6 address: fc00:f853:ccd:e793::2
[!!!!!!] Failed to mount API filesystems, freezing.
```

**What you expected to happen**:

Cluster cluster creation completes successfully

**How to reproduce it (as minimally and precisely as possible)**:

1. create the following `Vagrantfile`
1. `vagrant up`
1. due to the embedded `while` loop in the provisioning script, there will be a separate process that emits the logs of the `eks0-eks-a-cluster` container for your convenience

```ruby
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/impish64"
config.vm.network "private_network", ip: "192.168.33.105"

config.vm.provider "virtualbox" do |vb|
vb.memory = "4098"
end
config.vm.provision "shell", inline: <<-SHELL
set -ex
export DEBIAN_FRONTEND=noninteractive
if ! command -v gnupg &>/dev/null; then
apt-get update

apt-get install -y \
ca-certificates \
curl \
gnupg \
lsb-release
fi
krf=/usr/share/keyrings/docker-archive-keyring.gpg
if [ ! -e $krf ]; then
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o $krf
fi
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=$krf] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" >/etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io

docker version

if [ ! -e /usr/local/bin/eksctl ]; then
if [ ! -e /tmp/eksctl ]; then
curl -fsSL "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" \
| tar -xz -C /tmp
fi
cp /tmp/eksctl /usr/local/bin
fi
if [ ! -e /usr/local/bin/eksctl-anywhere ]; then
EKSA_RELEASE="0.6.0" OS="$(uname -s | tr A-Z a-z)" RELEASE_NUMBER=2
curl -fsSL "https://anywhere-assets.eks.amazonaws.com/releases/eks-a/${RELEASE_NUMBER}/artifacts/eks-a/v${EKSA_RELEASE}/${OS}/eksctl-anywhere-v${EKSA_RELEASE}-${OS}-amd64.tar.gz" \
| tar -xzC /usr/local/bin ./eksctl-anywhere
fi

eksctl anywhere version
# since the images are huge, load them if available
cd ~vagrant
for i in /vagrant/*.tar ./*.tar; do
[ -e "$i" ] || continue
docker load -i "$i"
done
eksctl anywhere generate clusterconfig eks0 --provider docker --verbosity 10000 > eks0.yaml 2>eks0.log
bash -c 'while true; do cid=$(docker ps -qf label=io.x-k8s.kind.role=control-plane || true); if [ -z "$cid" ]; then sleep 2; continue; fi; exec docker logs -f $cid; done' &
eksctl anywhere create cluster -f eks0.yaml --verbosity 10000 2>&1| tee -a eks0.log
SHELL
end
```

**Anything else we need to know?**:

A cursory search for that error message made it seem it is related to trying to run systemd inside docker: https://stackoverflow.com/questions/64349278/unable-to-start-systemd-container-using-docker-centos-7-8-host-failed-to-moun

**Environment**:
- EKS Anywhere Release: v0.6.0
- EKS Distro Release: unknown, presumably `v1.21.2-eks-d-1-21-6-eks-a-4`

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the provided Vagrantfile and the EKS Anywhere Docker provider, then compare eks0.log with the control-plane container logs showing the mount error. Trace the bootstrap-cluster creation path from the reported command and determine what must change for cluster creation to complete successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.