aws / aws/res

[BUG] Changing a GPU desktop's instance type across GPU families leaves the session in ERROR

Open
#178 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
115
Forks
35
PR merge metrics
No merged PRs in 30d

Description

Changing an existing GPU virtual desktop's instance type to a different GPU family
via the RES UI, for example g4dn/T4 to g6/L4, leaves the session in ERROR after
RESUMING/INITIALIZING. The instance starts and the GPU driver and DCV server come
up, but the DCV console session's X server never initializes because
`/etc/X11/xorg.conf` is still pinned to the create-time GPU's PCI BusID. The host
bootstrap does re-run on every boot, but the X-config step self-skips when a driver
is already present, so the stale BusID is never refreshed. Fresh desktops created
directly on the target type reach READY, and CPU-only type changes reach READY, so
the issue is specific to GPU-family changes on existing desktops. Reproduced both
g4dn to g6 and g6 to g4dn.

## Environment

- RES 2026.06
- Base OS: Ubuntu 24.04 and Amazon Linux 2023. The Linux GPU path is OS-agnostic; see Root cause.
- GPU instances: g4dn (T4), g6 (L4); NVIDIA GRID driver 595.71.05 from `s3://ec2-linux-nvidia-drivers/latest/`
- Region: us-east-1

## Steps to reproduce

1. Create a RES Linux GPU desktop on `g4dn.2xlarge`. It reaches READY.
2. Stop the desktop.
3. Change its instance type to a different GPU family, e.g. `g6.2xlarge`.
4. Resume the desktop.

## Expected

The desktop resumes on the new GPU type and reaches READY, as a fresh create on that
type does.

## Actual

The session goes RESUMING to INITIALIZING to ERROR. On the host the instance is
running, `nvidia-smi` reports the new GPU and driver, and `dcvserver` is active, but
no X server is running and the DCV console session never initializes.

## Root cause

RES bakes the GPU family into the host bootstrap's `install.sh -g ` argument
at create time, computed from the create-time instance type. The Linux userdata
cloud-config runs on every boot via `cloud_final_modules: [scripts-user, always]`,
and an instance-type change (`ModifyInstanceAttribute`) followed by resume
(`StartInstances`) does not rebuild the userdata, so on resume the instance re-runs
the create-time bootstrap.

For a GPU-family change the re-run does not refresh the GPU X configuration.
`install_nvidia_grid_drivers` in `gpu_drivers.sh` short-circuits on `which
nvidia-smi` — the pre-existing GRID driver satisfies the check — and returns before
running `nvidia-xconfig --preserve-busid`, the step that writes
`/etc/X11/xorg.conf` pinned to the GPU's PCI BusID. So the xorg.conf generated for
the original GPU is never regenerated for the new GPU: X cannot start on the new
GPU's BusID, the DCV console session never initializes, and RES marks the session
ERROR.

A fresh create reaches READY because no driver is present, so the guard passes and
`nvidia-xconfig` runs for the correct GPU. A CPU-only change reaches READY because
no GPU/X stack is involved.

Code (RES 2026.06):

- GPU family baked into `install.sh -g` at create time:
[virtual_desktop_controller_utils.py L93-L97, L152](https://github.com/aws/res/blob/2026.06/source/idea/idea-virtual-desktop-controller/src/ideavirtualdesktopcontroller/app/virtual_desktop_controller_utils.py#L152)
- Userdata cloud-config runs on every boot:
[bootstrap_userdata_linux.sh.jinja2 L11-L12](https://github.com/aws/res/blob/2026.06/source/idea/library/src/res/utils/_templates/linux/bootstrap_userdata_linux.sh.jinja2#L11-L12)
- Type change performs only `ModifyInstanceAttribute`; resume performs only `StartInstances`; userdata is not rebuilt:
[virtual_desktop_controller_utils.py L727-L742](https://github.com/aws/res/blob/2026.06/source/idea/idea-virtual-desktop-controller/src/ideavirtualdesktopcontroller/app/virtual_desktop_controller_utils.py#L727-L742),
[virtual_desktop_server_utils.py L100-L113](https://github.com/aws/res/blob/2026.06/source/idea/idea-virtual-desktop-controller/src/ideavirtualdesktopcontroller/app/servers/virtual_desktop_server_utils.py#L100-L113)
- X-config step self-skips on `which nvidia-smi`, so `nvidia-xconfig --preserve-busid` never re-runs after a GPU change; skip at L34-L36, `nvidia-xconfig` at L63:
[gpu_drivers.sh L33-L64](https://github.com/aws/res/blob/2026.06/source/idea/idea-bootstrap/resources/scripts/dcv/linux/gpu_drivers.sh#L33-L64)

`gpu_drivers.sh` branches by instance family, not OS, and is shared by Ubuntu,
Amazon Linux, and RHEL through the DCV bootstrap, so all Linux distros are expected
to be affected.

## Evidence

Same AMI, driver, and kernel across all cases.

- g4dn -> g6: ERROR. Instance running, `nvidia-smi` shows the L4, `dcvserver` active, but no "First frame captured" and `dcvgltest` returns "Unable to open display". Started immediately (no `InsufficientInstanceCapacity`) and DCV-ready ~2.4 min after start, still ERROR — rules out capacity and readiness-timeout.
- g6 -> g4dn: ERROR. `xorg.conf` = `BusID "PCI:49:0:0"` / `BoardName "NVIDIA L4"`; live GPU `Tesla T4` at `00000000:00:1E.0` (bus 0x1E) — BusID mismatch. `pgrep Xorg`/`Xdcv` empty, no `/var/log/Xorg.0.log`; X never started.
- Fresh create on g4dn -> READY. Fresh create on g6 -> READY. Isolates the failure to the type-change path.
- CPU-only c8g -> c9g (Amazon Linux 2023) -> READY. Scopes the bug to GPU desktops.

## Suggested fix

Refresh the GPU X configuration when the GPU changes. Options:

- On boot, detect a GPU change (current PCI BusID or GPU model differs from what `/etc/X11/xorg.conf` was generated for) and re-run `nvidia-xconfig --preserve-busid --enable-all-gpus` even when the driver is already installed. Do not gate the X-config step behind the `which nvidia-smi` driver-presence check.
- Or force host GPU/DCV/X re-provisioning on the next resume after a GPU-family change.
- Or block cross-GPU-family instance-type changes on an existing desktop and direct the user to recreate on the target type.

## Workaround

Recover an already-ERROR desktop in place, validated. On the host as root:

```
rm -f /etc/X11/xorg.conf
nvidia-xconfig --enable-all-gpus
# confirm the BusID now matches `nvidia-smi --query-gpu=pci.bus_id`
```

Then use the RES Reboot action, the only action available from ERROR. It sets the
session to PROVISIONING and re-creates the DCV session. The corrected `xorg.conf`
persists on the EBS root volume and the `which nvidia-smi` skip guard leaves it
untouched, so X starts on the correct GPU and the session reaches READY. Verified:
the desktop returned to READY after the reboot and remained READY across a
subsequent stop/start.

Alternatively, recreate the desktop on the target instance type. A fresh create runs
the full create-time bootstrap and configures the GPU display stack for the correct
GPU. User data on shared home storage is preserved.

Contributor guide

Open the contributing guide

Research direction

Start with gpu_drivers.sh, especially the driver-presence guard and nvidia-xconfig step, then trace the instance-type change and resume flow in virtual_desktop_controller_utils.py and virtual_desktop_server_utils.py. Reproduce a g4dn-to-g6 or g6-to-g4dn change and inspect the generated /etc/X11/xorg.conf. Done means an existing desktop resumes on the new GPU family and reaches READY with an X configuration matching the active GPU.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, linux, python, shell
Domain
desktop, infrastructure, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.