Problems netbooting non-EFI Pi 4 into HookOS

Hello Tinkerbell team,

I'm currently trying to get HookOS netbooted on a Raspberry Pi 4 without EFI.

Setup

All of my attempts are working with a Dnsmasq setup, with Tinkerbell's DHCP functionality completely switched off. Dnsmasq is supplying the Pi boot files via TFTP. For all of the Raspberry Pi firmware files, I've downloaded the Armbian image from here, specifically the "Minimal/IOT images with Armbian Linux v6.12" Ubuntu Noble image. I then copied the entire content into the TFTP root directory.

The config.txt I used did not change much, besides sometimes changing the kernel/initramfs filenames:

# Automatically load overlays for detected cameras
camera_auto_detect=1

# Automatically load overlays for detected DSI displays
display_auto_detect=1

# Automatically load initramfs files, if found
# auto_initramfs=1

# Don't have the firmware create an initial video= setting in cmdline.txt.
# Use the kernel's default instead.
disable_fw_kms_setup=1

# Disable compensation for displays with overscan
disable_overscan=1

# Run as fast as firmware / board allows
arm_boost=1

[pi4]
max_framebuffer=2

[cm4]
# Enable host mode on the 2711 built-in XHCI USB controller.
# This line should be removed if the legacy DWC2 controller is required
# (e.g. for USB device mode) or if USB support is not required.
otg_mode=1

[cm5]
dtoverlay=dwc2,dr_mode=host

[all]
kernel=vmlinuz-armbian-bcm2711-legacy
initramfs initramfs-armbian-bcm2711-legacy followkernel
arm_64bit=1
cmdline=cmdline.txt
dtoverlay=dwc2
dtoverlay=vc4-kms-v3d
enable_uart=1

The cmdline.txt did change a bit for each test, but they all had the same result. I copied it from a successful EFI/iPXE boot on the same Pi to get all the correct Tinkerbell parameters. One example:

tink_worker_image=ghcr.io/tinkerbell/tink-agent:v0.18.3-b817f7f2 syslog_host=192.0.2.111 grpc_authority=192.0.2.111:42113 tinkerbell_tls=false tinkerbell_insecure_tls=false worker_id=e4:5f:01:bc:f4:ce hw_addr=e4:5f:01:bc:f4:ce modules=loop,squashfs,sd-mod,usb-storage console=serial0,115200

Attempt with vanilla HookOS 0.10

My first boot attempt was with the initramfs and kernel from the HookOS 0.10.0 release, specifically the hook_armbian-bcm2711-current.tar.gz file. This attempt did successfully boot the kernel and the initramfs, as evidenced by the fact that containerd got started. There were a couple of errors in the logs for that attempt:

Failed to read service spec error "open /containers/services/getty/config.json: No such file or directory"

That was the last message. Past that, no further errors appeared and I wasn't getting a console (understandable, because getty failed to start).
Looking a bit further up, I also saw this message:

failed to run hwclock: exit status 1

That appears right after the kernel is done booting and right before the LinuxKit splash appears with "Welcome to LinuxKit". This error was confirmed by the next log lines from the containerd startup, which all had 1970-01-01 timestamps. This error makes sense of course, because the Pi 4 doesn't have a HW clock.

Scrolling through the kernel logs, the only problematic message I was able to see was this one:

rootfs image is not initramfs (read error): looks like an initrd

I'm not sure whether that one is really relevant though, because the initramfs is definitely getting mounted. Just something seems to be wrong.

Going back to the initial message, I thought something was wrong with the initramfs and I checked the following:

• The /containers/services/getty/config.json file is present in the initramfs
• The initramfs itself seems to be fine, unpacks without issue running gunzip -c initramfs | cpio -i
• Repacked the initramfs, this then lead to a kernel panic - I don't know what I might have done wrong, I ran find . | cpio -H newc -o | gzip -9 > /somepath/initramfs

I tried a lot of other things, especially changing the cmdline.txt, but with no effect at all. Amongst other things, I tried an Ubuntu 24.04 kernel I'm using for my production Pi 4, it showed exactly the same behaviour.

Attempt with the most current armbian-bcm2711-current

Next I wanted to check whether a newer kernel would work. I cloned the Hook repo at commit 0182cbca7ef06bf9fa6027b1249a99ac9b007e0e, the current HEAD of main.

I then ran then following commands:

./build.sh kernel armbian-bcm2711-current
./build.sh build armbian-bcm2711-current

When I executed that command, it provided me with 6.12.35-S8292-Dbdda-P0000-Ce6dbH2313-HK01ba-Vc222-Ba566-R448a.

I copied the kernel and initramfs as well as the overlays/dtbs produced into the TFTP root and booted. Just to be greeted by a similar issue. The only difference was that I wasn't getting the previous getty startup error. Instead, there was just this message as the only error I could find:

WARN[1970-01-01T00:00:17.287409971Z] failed to load plugin                         error="failed to create CRI service: failed to create cni conf monitor for default: failed to create the parent of the cni conf dir=/etc/cni: mkdir /etc/cn
i: read-only file system" id=io.containerd.grpc.v1.cri type=io.containerd.grpc.v1

The hwclock issue also persisted. At the end of the log output, I got this message:

INFO[1970-01-01T00:00:17.289087879Z] containerd successfully booted in 1.073162s

And then nothing more. As before, I did not get a console prompt.

Attempt with a 5.15 Armbian kernel

Next, I tried an older 5.15 Armbian kernel, just on the odd chance that it had something to do with the newer kernel. I introduced the following change in to the bash/inventory.sh file:

      #
      ## Armbian bcm2711 (Broadcom) legacy, from RaspberryPi Foundation with many CNCF-landscape fixes and patches; for the RaspberryPi 3b+/4b
      define_id "armbian-bcm2711-legacy" METHOD='armbian' ARCH='aarch64' TAG='armbian-sbc' ARMBIAN_KERNEL_ARTIFACT='kernel-bcm2711-legacy'
      add_bootable_id "rpi" HANDLER='rpi_firmware' TAG='armbian-sbc'

Then I rebuild HookOS:

export ARMBIAN_KERNEL_VERSION="5.15.92-S14b3-Dfae2-P3c78-C7e7bHfe66-HK01ba-Vc222-B6a41-R448a"
./build.sh kernel armbian-bcm2711-legacy
./build.sh build armbian-bcm2711-legacy

That kernel version was the newest non-6.0 kernel I could find.

Booting this kernel/initramfs showed exactly the same issue as above. Containerd shows a happy "containerd booted successfully", and the nothing at all further happens.

Final

During all of these boots, the Pi never acquired an IP. As far as I could see that's because DHCPC is launched in a container as well. I also saw absolutely nothing in the Tinkerbell logs, which makes sense considering that the Pi never got an IP.

If there's anything more I can provide, please ping me.

Reproduce the non-EFI Raspberry Pi 4 boot using the supplied Dnsmasq/TFTP setup and compare the HookOS 0.10 and current Armbian attempts. Read bash/inventory.sh and the rpi_firmware handler, then inspect the kernel, initramfs, containerd, and network logs; done means the Pi obtains an IP and reaches a usable console with Tinkerbell activity.