hashicorp / hashicorp/packer-plugin-amazon

Amazon chroot builder: Allow nvme-capable instances w/ proper udev rules to use symlinks under /dev

Open
#24 1 comment 1 reaction 0 assignees View on GitHub
builder/amazon enhancement
Dominant language
Go
Stars
91
Forks
141
Avg merge
2d 18h
Merged PRs (30d)
3

Description

_This issue was originally opened by @bmoyles as hashicorp/packer#10807. It was migrated here as a result of the [Packer plugin split](https://github.com/hashicorp/packer/issues/8610#issuecomment-770034737). The original body of the issue is below._


#### Description

When NVMe-capable instances were released, the Amazon chroot builder was changed to take an explicit NVMe path to the real NVMe device and that is checked against the path under `/sys/block`. Amazon Linux (and some other distributions) ship with tooling and udev rules that look in the attached NVMe block device's metadata for the block device name that was passed to the EC2 API on attach and set up the appropriate symlinks under `/dev/` to the real NVMe device (which is fantastic as the NVMe device order is *not* guaranteed).

It would be fantastic (and I would be happy to submit a PR if no one is able to take this on) if we could provide a configuration option to the chroot builder that indicates it should look under `/dev` for a symlink to the block device based on the device in the attach request. eg if one asks for `vol-deadbeef` to be attached to `/dev/xvdg` in the attach request, there will be a `/dev/xvdg` symlink pointing to the real block device.

(A step further might be to have Packer itself use `nvme id-ctrl` or some Golang-native equivalent do the inspection of the NVMe devices to find the requested volume which would allow this to work on distributions w/o the necessary udev configuration, but that's a much bigger change and I think simply indicating that this NVME-capable instance will have symlinks for Packer is sufficient for a first-pass).

#### Use Case(s)

At Netflix, we have multiple instances baking images in parallel (with multiple processes per-instance) which makes the NVMe device much less predictable, but we DO have the udev configuration rolled into our base images such that we always have the correct symlink for EBS volumes on NVMe instances (whether the request was made for xvd* or sd* -- we create both symlinks to eliminate confusion).

#### Potential configuration
I might approach this by simply allowing one to explicitly specify their device-prefix-of-choice via configuration which would short-circuit the code in `devicePrefix` https://github.com/hashicorp/packer/blob/master/builder/amazon/chroot/device.go#L15 to avoid the scan of `/sys/block` for the block device. This, to me, would indicate that I am asserting my device, when attached, will have a file (symlink) under /dev/ with the appropriate prefix + letter (and/or partition as necessary) so just look for the presence/absence of those links under /dev:

```
block-device-prefix: xvd
# or block-device-prefix: sd
```
Alternatively, we could go down a different path altogether with explicit configuration such as
```
nvme-device-symlinks-enabled: true
```
signaling that a) we're on a NVMe-enabled instance, b) if we request the device be attached to /dev/xvd* or /dev/sd*, there will be a symlink created when the device is attached, and c) presence or absence of those symlinks is sufficient to determine whether or not our block device is present.

I could approach this via a custom plugin that duplicates the chroot builder and extends the functionality internally, but this seems generally-applicable so figured I'd pose it here before going down that route. If I went that route, I personally would probably lean towards allowing the user to explicitly declare the block-device prefix as it seems like the route with the least moving pieces.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.