Raspberry Pi 5 – unstable system with multiple NVME

Start by reproducing the failure with Raspberry Pi 5, multiple NVMe disks, Debian Trixie, and kernel 6.12.75+rpt-rpi-2712, then compare behavior with kernel 6.6.74+rpt-rpi-2712. Review the reported D3hot-to-D0 errors and test the listed APST, ASPM, and PCIe Gen2 settings; done means the regression is explained and drive dropouts no longer occur under the reported workload.

Describe the bug

Hello,
I’m seeing stability issues with NVMe SSDs on Raspberry Pi 5 and would like to know if others experience the same.
Also i am proposin my solution.

Setup:
Raspberry Pi: Raspberry Pi 5
Raspberry Pi OS (Debian Trixie, Kernel 6.12.75+rpt-rpi-2712)

NVMe hub: X1016 NVMe shield (power supply 9V - 17V) - this shield also supplying Pi5 through pogo pins
SSDs:
2× WD Red SN700 NVMe 2 TB (RAID array)
1× Apacer AS2280P4U Pro 256 GB (system disk)

Power supply: my final ( i tried multiple of them ) 15.5 V / 8A (max 40mV peak2peak)

Issuue description
RAID built from 2× WD Red SN700 becomes unstable
One WD drive intermittently drops out
RAID goes degraded / fails
Apacer drive is usually unaffected

Kernel log repeatedly shows:

• pcieport XXXX:XX:XX.X: unable to change power state from D3hot to D0, device inaccessible
• randomly dissapearing one or more discs during boot or during using them (if boot by chances passed)
• Triggered during I/O load, RAID activity, or NVMe power state transitions.

Notes
X1016 vendor lists only Samsung PM961 as incompatible
WD Red SN700 and Apacer are not listed as incompatible
Disks are NVMe (not SATA / AHCI)

My first suspect cause:
Likely a power / power‑management interaction:
aggressive NVMe power states (APST / D3hot)
high peak current of WD Red SN700 (two drives in parallel)
shared 3.3 V rail on the hub
RPi 5 PCIe + PCIe switch
12 V / 5 A power supply likely borderline - then i used even better power resources (listed at beginning) - this was not a root cause

What have I done during debug

• Disable NVMe low‑power states:
  ‘nvme_core.default_ps_max_latency_us=0’
• Disable PCIe ASPM:
  ‘pcie_aspm=off’
• Force PCIe Gen2 for stability:
  ‘INIdtparam=pciex1_gen=2’
• Use a stronger PSU:
  12 V / 8–10 A recommended; then I used even better source -> check the beginning

Final Fix

• revert verison OS/kernel from (Debian Trixie, Kernel 6.12.75+rpt-rpi-2712) to
  Kernel 6.6.74+rpt-rpi-2712

It is a few day, everything looks perfect.

Steps to reproduce the behaviour

use (Debian Trixie, Kernel 6.12.75+rpt-rpi-2712) and multiple NVME disks (I am not sure if x1016 is important in this chain)

Device (s)

Raspberry Pi 5

System

Debian Trixie, Kernel 6.12.75+rpt-rpi-2712

Logs

• pcieport XXXX:XX:XX.X: unable to change power state from D3hot to D0, device inaccessible
• randomly dissapearing one or more discs during boot or during using them (if boot by chances passed)
• Triggered during I/O load, RAID activity, or NVMe power state transitions.

Additional context

Final Fix

• revert verison OS/kernel from (Debian Trixie, Kernel 6.12.75+rpt-rpi-2712) to
  Kernel 6.6.74+rpt-rpi-2712