[BUG] Changing a GPU desktop's instance type across GPU families leaves the session in ERROR
- Dominant language
- Python
- Stars
- 115
- Forks
- 35
- PR merge metrics
- No merged PRs in 30d
Description
Changing an existing GPU virtual desktop's instance type to a different GPU family
via the RES UI, for example g4dn/T4 to g6/L4, leaves the session in ERROR after
RESUMING/INITIALIZING. The instance starts and the GPU driver and DCV server come
up, but the DCV console session's X server never initializes because
`/etc/X11/xorg.conf` is still pinned to the create-time GPU's PCI BusID. The host
bootstrap does re-run on every boot, but the X-config step self-skips when a driver
is already present, so the stale BusID is never refreshed. Fresh desktops created
directly on the target type reach READY, and CPU-only type changes reach READY, so
the issue is specific to GPU-family changes on existing desktops. Reproduced both
g4dn to g6 and g6 to g4dn.
## Environment
- RES 2026.06
- Base OS: Ubuntu 24.04 and Amazon Linux 2023. The Linux GPU path is OS-agnostic; see Root cause.
- GPU instances: g4dn (T4), g6 (L4); NVIDIA GRID driver 595.71.05 from `s3://ec2-linux-nvidia-drivers/latest/`
- Region: us-east-1
## Steps to reproduce
1. Create a RES Linux GPU desktop on `g4dn.2xlarge`. It reaches READY.
2. Stop the desktop.
3. Change its instance type to a different GPU family, e.g. `g6.2xlarge`.
4. Resume the desktop.
## Expected
The desktop resumes on the new GPU type and reaches READY, as a fresh create on that
type does.
## Actual
The session goes RESUMING to INITIALIZING to ERROR. On the host the instance is
running, `nvidia-smi` reports the new GPU and driver, and `dcvserver` is active, but
no X server is running and the DCV console session never initializes.
## Root cause
RES bakes the GPU family into the host bootstrap's `install.sh -g ` argument
at create time, computed from the create-time instance type. The Linux userdata
cloud-config runs on every boot via `cloud_final_modules: [scripts-user, always]`,
and an instance-type change (`ModifyInstanceAttribute`) followed by resume
(`StartInstances`) does not rebuild the userdata, so on resume the instance re-runs
the create-time bootstrap.
For a GPU-family change the re-run does not refresh the GPU X configuration.
`install_nvidia_grid_drivers` in `gpu_drivers.sh` short-circuits on `which
nvidia-smi` — the pre-existing GRID driver satisfies the check — and returns before
running `nvidia-xconfig --preserve-busid`, the step that writes
`/etc/X11/xorg.conf` pinned to the GPU's PCI BusID. So the xorg.conf generated for
the original GPU is never regenerated for the new GPU: X cannot start on the new
GPU's BusID, the DCV console session never initializes, and RES marks the session
ERROR.
A fresh create reaches READY because no driver is present, so the guard passes and
`nvidia-xconfig` runs for the correct GPU. A CPU-only change reaches READY because
no GPU/X stack is involved.
Code (RES 2026.06):
- GPU family baked into `install.sh -g` at create time:
[virtual_desktop_controller_utils.py L93-L97, L152](https://github.com/aws/res/blob/2026.06/source/idea/idea-virtual-desktop-controller/src/ideavirtualdesktopcontroller/app/virtual_desktop_controller_utils.py#L152)
- Userdata cloud-config runs on every boot:
[bootstrap_userdata_linux.sh.jinja2 L11-L12](https://github.com/aws/res/blob/2026.06/source/idea/library/src/res/utils/_templates/linux/bootstrap_userdata_linux.sh.jinja2#L11-L12)
- Type change performs only `ModifyInstanceAttribute`; resume performs only `StartInstances`; userdata is not rebuilt:
[virtual_desktop_controller_utils.py L727-L742](https://github.com/aws/res/blob/2026.06/source/idea/idea-virtual-desktop-controller/src/ideavirtualdesktopcontroller/app/virtual_desktop_controller_utils.py#L727-L742),
[virtual_desktop_server_utils.py L100-L113](https://github.com/aws/res/blob/2026.06/source/idea/idea-virtual-desktop-controller/src/ideavirtualdesktopcontroller/app/servers/virtual_desktop_server_utils.py#L100-L113)
- X-config step self-skips on `which nvidia-smi`, so `nvidia-xconfig --preserve-busid` never re-runs after a GPU change; skip at L34-L36, `nvidia-xconfig` at L63:
[gpu_drivers.sh L33-L64](https://github.com/aws/res/blob/2026.06/source/idea/idea-bootstrap/resources/scripts/dcv/linux/gpu_drivers.sh#L33-L64)
`gpu_drivers.sh` branches by instance family, not OS, and is shared by Ubuntu,
Amazon Linux, and RHEL through the DCV bootstrap, so all Linux distros are expected
to be affected.
## Evidence
Same AMI, driver, and kernel across all cases.
- g4dn -> g6: ERROR. Instance running, `nvidia-smi` shows the L4, `dcvserver` active, but no "First frame captured" and `dcvgltest` returns "Unable to open display". Started immediately (no `InsufficientInstanceCapacity`) and DCV-ready ~2.4 min after start, still ERROR — rules out capacity and readiness-timeout.
- g6 -> g4dn: ERROR. `xorg.conf` = `BusID "PCI:49:0:0"` / `BoardName "NVIDIA L4"`; live GPU `Tesla T4` at `00000000:00:1E.0` (bus 0x1E) — BusID mismatch. `pgrep Xorg`/`Xdcv` empty, no `/var/log/Xorg.0.log`; X never started.
- Fresh create on g4dn -> READY. Fresh create on g6 -> READY. Isolates the failure to the type-change path.
- CPU-only c8g -> c9g (Amazon Linux 2023) -> READY. Scopes the bug to GPU desktops.
## Suggested fix
Refresh the GPU X configuration when the GPU changes. Options:
- On boot, detect a GPU change (current PCI BusID or GPU model differs from what `/etc/X11/xorg.conf` was generated for) and re-run `nvidia-xconfig --preserve-busid --enable-all-gpus` even when the driver is already installed. Do not gate the X-config step behind the `which nvidia-smi` driver-presence check.
- Or force host GPU/DCV/X re-provisioning on the next resume after a GPU-family change.
- Or block cross-GPU-family instance-type changes on an existing desktop and direct the user to recreate on the target type.
## Workaround
Recover an already-ERROR desktop in place, validated. On the host as root:
```
rm -f /etc/X11/xorg.conf
nvidia-xconfig --enable-all-gpus
# confirm the BusID now matches `nvidia-smi --query-gpu=pci.bus_id`
```
Then use the RES Reboot action, the only action available from ERROR. It sets the
session to PROVISIONING and re-creates the DCV session. The corrected `xorg.conf`
persists on the EBS root volume and the `which nvidia-smi` skip guard leaves it
untouched, so X starts on the correct GPU and the session reaches READY. Verified:
the desktop returned to READY after the reboot and remained READY across a
subsequent stop/start.
Alternatively, recreate the desktop on the target instance type. A fresh create runs
the full create-time bootstrap and configures the GPU display stack for the correct
GPU. User data on shared home storage is preserved.
Contributor guide
Research direction
Start with gpu_drivers.sh, especially the driver-presence guard and nvidia-xconfig step, then trace the instance-type change and resume flow in virtual_desktop_controller_utils.py and virtual_desktop_server_utils.py. Reproduce a g4dn-to-g6 or g6-to-g4dn change and inspect the generated /etc/X11/xorg.conf. Done means an existing desktop resumes on the new GPU family and reaches READY with an X configuration matching the active GPU.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, linux, python, shell
- Domain
- desktop, infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100