cluster creation fails with "Failed to mount API filesystems, freezing"
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**:
```console
$ eksctl anywhere generate clusterconfig eks0 --provider docker --verbosity 10000 > eks0.yaml 2>eks0.log
$ eksctl anywhere create cluster -f eks0.yaml --verbosity 10000 2>&1| tee -a eks0.log
...
Error: failed to create cluster: error creating bootstrap cluster: error executing create cluster: exit status 1, try rerunning with --force-cleanup to force delete previously created bootstrap cluster
```
Only by looking separately at the `` container logs does one see the actual failure:
```console
$ docker logs $(docker ps -qf label=io.x-k8s.kind.role=control-plane)
INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v2
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from VM UUID.
INFO: faking /sys/class/dmi/id/product_name to be "kind"
INFO: faking /sys/class/dmi/id/product_uuid to be random
INFO: faking /sys/devices/virtual/dmi/id/product_uuid as well
INFO: setting iptables to detected mode: legacy
INFO: Detected IPv4 address: 172.18.0.2
INFO: Detected IPv6 address: fc00:f853:ccd:e793::2
[!!!!!!] Failed to mount API filesystems, freezing.
```
**What you expected to happen**:
Cluster cluster creation completes successfully
**How to reproduce it (as minimally and precisely as possible)**:
1. create the following `Vagrantfile`
1. `vagrant up`
1. due to the embedded `while` loop in the provisioning script, there will be a separate process that emits the logs of the `eks0-eks-a-cluster` container for your convenience
```ruby
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/impish64"
config.vm.network "private_network", ip: "192.168.33.105"
config.vm.provider "virtualbox" do |vb|
vb.memory = "4098"
end
config.vm.provision "shell", inline: <<-SHELL
set -ex
export DEBIAN_FRONTEND=noninteractive
if ! command -v gnupg &>/dev/null; then
apt-get update
apt-get install -y \
ca-certificates \
curl \
gnupg \
lsb-release
fi
krf=/usr/share/keyrings/docker-archive-keyring.gpg
if [ ! -e $krf ]; then
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o $krf
fi
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=$krf] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" >/etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io
docker version
if [ ! -e /usr/local/bin/eksctl ]; then
if [ ! -e /tmp/eksctl ]; then
curl -fsSL "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" \
| tar -xz -C /tmp
fi
cp /tmp/eksctl /usr/local/bin
fi
if [ ! -e /usr/local/bin/eksctl-anywhere ]; then
EKSA_RELEASE="0.6.0" OS="$(uname -s | tr A-Z a-z)" RELEASE_NUMBER=2
curl -fsSL "https://anywhere-assets.eks.amazonaws.com/releases/eks-a/${RELEASE_NUMBER}/artifacts/eks-a/v${EKSA_RELEASE}/${OS}/eksctl-anywhere-v${EKSA_RELEASE}-${OS}-amd64.tar.gz" \
| tar -xzC /usr/local/bin ./eksctl-anywhere
fi
eksctl anywhere version
# since the images are huge, load them if available
cd ~vagrant
for i in /vagrant/*.tar ./*.tar; do
[ -e "$i" ] || continue
docker load -i "$i"
done
eksctl anywhere generate clusterconfig eks0 --provider docker --verbosity 10000 > eks0.yaml 2>eks0.log
bash -c 'while true; do cid=$(docker ps -qf label=io.x-k8s.kind.role=control-plane || true); if [ -z "$cid" ]; then sleep 2; continue; fi; exec docker logs -f $cid; done' &
eksctl anywhere create cluster -f eks0.yaml --verbosity 10000 2>&1| tee -a eks0.log
SHELL
end
```
**Anything else we need to know?**:
A cursory search for that error message made it seem it is related to trying to run systemd inside docker: https://stackoverflow.com/questions/64349278/unable-to-start-systemd-container-using-docker-centos-7-8-host-failed-to-moun
**Environment**:
- EKS Anywhere Release: v0.6.0
- EKS Distro Release: unknown, presumably `v1.21.2-eks-d-1-21-6-eks-a-4`
Contributor guide
Research direction
Reproduce the failure with the provided Vagrantfile and the EKS Anywhere Docker provider, then compare eks0.log with the control-plane container logs showing the mount error. Trace the bootstrap-cluster creation path from the reported command and determine what must change for cluster creation to complete successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100