Persistent disk mount issue with Nitro instances

Summary

On AWS Nitro-based instances, attaching a persistent disk and deploying fails with:

mount /dev/nvme2n1p1 /var/vcap/store -> wrong fs type, bad superblock

BOSH then detaches the disk and rolls back. The VM is left with only root and ephemeral
disks.

Environment

• AWS Nitro-based instance
• Agent config at time of failure:

  "disks": {
    "system": "/dev/xvda",
    "ephemeral": "/dev/sdb",
    "persistent": {}
  }
  
  

Observed behaviour

• Persistent disk is created and attached successfully by the CPI
• Agent resolves the device path and attempts to mount it
• Mount fails: wrong fs type, bad superblock
• BOSH detaches the disk and rolls back

Analysis

persistent is empty in the agent settings at mount time, meaning the agent resolves
the device path dynamically via IDDevicePathResolver. On Nitro instances, NVMe PCIe
enumeration order is non-deterministic. If the symlink in /dev/disk/by-id/ resolves to
the wrong NVMe device — one that is unformatted or has the wrong filesystem — the mount
fails with bad superblock.

The disk ID passed from the CPI may not match the symlink name, or there may be multiple
matching symlinks causing ambiguity.

Related

• cloudfoundry/bosh-agent#407 — fixes the same class of problem for NVMe instance
  storage discovery
• cloudfoundry/bosh-aws-cpi-release#196 — companion CPI-side change

Co-authored with Claude Code

Start with the IDDevicePathResolver and persistent-disk device-path handling, then compare the behavior with cloudfoundry/bosh-agent#407 and cloudfoundry/bosh-aws-cpi-release#196. Reproduce the Nitro-instance deployment described here and verify that the attached disk resolves to the correct NVMe device and mounts successfully without rollback.