Raspberry Pi 4 : USB3 SSD Connected via USB3 Hub : BOOT Files Corrupted Following Reboot

Start with the attached test.zip and its test script, using the listed Raspberry Pi 4, USB3 hub, SSD, and Raspberry Pi OS Bullseye setup. Reproduce the copy, diff, reboot, and post-reboot diff sequence, then investigate the kernel path involved in USB3 storage and FAT32 writes. Done means the corruption is explained and the reproduction no longer leaves /boot files corrupted.

Describe the bug

Files written to the FAT32 BOOT filesystem of a USB3 SSD connected via a USB3 hub sporadically get corrupted when followed immediately by a reboot. This issue does not occur when the SSD is connected directly to one of the Raspberry Pi 4 USB3 ports. This issue only occurs on the FAT32 BOOT filesystem, never on the EXT4 ROOT filesystem. A 'sync' command prior to the reboot does not eliminate the issue. This issue does not occur if the hub is connected to one of the Raspberry Pi 4 USB2 ports.

Steps to reproduce the behaviour

Test environment:

1. Raspbberry Pi 4B 4GB (revision 1.1 or 1.2)

2. SSD: Samsung EVO 860 or Corsair Neutron

3. SATA to USB3 adapter:
   Asmedia-based (from 5 different manufacturers) [Bus 002 Device 003: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge]
   or
   Seagate [Bus 002 Device 003: ID 0bc2:50a0 Seagate RSS LLC FA GoFlex Desk]

4. Powered USB3 Hub:
   Realtek-based [Bus 002 Device 003: ID 0bda:0411 Realtek Semiconductor Corp. Hub / Bus 001 Device 004: ID 0bda:5411 Realtek Semiconductor Corp. RTS5411 Hub]
   or
   VIA Labs-based [Bus 002 Device 004: ID 2109:0817 VIA Labs, Inc. USB3.0 Hub / Bus 001 Device 005: ID 2109:2817 VIA Labs, Inc. USB2.0 Hub]

5. Rapberry Pi OS Bullseye booted on the SSD (not running on an SD card)

The attached 'test' script performs the following actions:

1. Copy the entire file structure from /boot to /SAVED-BOOT
2. Copy the entire file structure from /SAVED-BOOT to /boot
3. Run 'diff -r /SAVED-BOOT /boot' to ensure /boot is identical to /SAVED-BOOT
4. Reboot

Following the reboot, executing a 'diff -r /SAVED-BOOT /boot' sporadically reveals that one or more /boot files are corrupt (frequency and degree of corruption varies).
test.zip

Device (s)

Raspberry Pi 4 Mod. B

System

Raspberry Pi reference 2021-10-30
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 288b21fc27e128ea6b330777aca68e0061ebf4fe, stage2

Jan 20 2022 13:56:48
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)

Linux raspberrypi 5.10.92-v7l+ #1514 SMP Mon Jan 17 17:38:03 GMT 2022 armv7l GNU/Linux

Logs

No response

Additional context

No response