Diagnosing unreliable reboots

Start by reading the linked issue 1559 and the diagnostic procedure in this issue, including /boot/cmdline.txt, /boot/config.txt, and /usr/lib/systemd/system-shutdown/debug.sh. The payload does not identify a code location, failing test, or concrete change; a maintainer would need to define whether the next step is kernel investigation or documentation work and what evidence would count as done.

This is a continuation of https://github.com/raspberrypi/firmware/issues/1559, a thread which is so long that the important diagnostic questions have got lost in the middle.

https://github.com/raspberrypi/firmware/issues/1559 was specifically about Pi 4, but the methods for diagnosing the problem are common to all Pis.

I'll be using the word "reboot" throughout this list, but it should apply equally to powering off.

1. When rebooting hangs, wait several minutes to see if it unsticks eventually.
2. Does sync && sudo reboot -f hang or reboot?
3. If reboot is very slow but eventually completes,
   1. Add systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M enforcing=0 to /boot/cmdline.txt
   2. Put the following in /usr/lib/systemd/system-shutdown/debug.sh:

      #!/bin/sh
      mount -o remount,rw /
      dmesg > /shutdown-log.txt
      mount -o remount,ro /
      reboot
      

   3. Make it executable:

      $ sudo chmod +x /usr/lib/systemd/system-shutdown/debug.sh
      

   4. Reboot twice - once to activate the logging, and once to generate /shutdown-log.txt.
   5. Upload /shutdown-log.txt.
   6. Remove the additions to cmdline.txt and delete (or make non-executable) /usr/lib/systemd/system-shutdown/debug.sh
4. If reboot doesn't complete, and if you have a serial/UART cable you can connect to header pins 6, 8, and 10, and a host computer (or another Raspberry Pi) to run a terminal app such as PuTTY or minicom:
   1. Add enable_uart=1 to /boot/config.txt.
   2. Add systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M enforcing=0 ignore_loglevel to /boot/cmdline.txt, being careful to remove quiet if it is present.
   3. Run sync to flush outstanding writes, then power cycle.
   4. Log in as usual (it will take a long time to boot, with thousands of lines of output, and if you are logging in over the serial connection you may have to hit Return to get the prompt), then sudo reboot and capture the output in your terminal program.