GPU access on Linux VMs?

In my Azure TRE deployment I am trying to give researchers access to GPU-based VMs running Linux. We're using the Standard_NV*ads_A10_v5 family, with NVIDIA A10 GPUs, and need to install the kernel drivers. I'm struggling to get this to work, and my question is, has anyone succeeded in doing this already, and if so how?

There are two routes to install the NVIDIA drivers in Ubuntu:

1. via the NVIDIA GPU Driver Extension from Azure, or
2. directly via Ubuntu admin tools.

Neither of these work well enough to function in the TRE.

The NVIDIA GPU Driver Extension will install correctly in a VM, but cannot be baked into a VM image, it seems it's lost in the act of generalising the VM. It could, in principle, be installed in the VM once it's running in the TRE, but that doesn't work because of the network isolation in the TRE. The extension attempts to install all sorts of stuff, including Thunderbird and Firefox via the snap store, but Nexus isn't configured for snap and the install fails. Apart from that, manually adding the extension isn't a nice solution, we don't generally allow our users access to the Azure portal to be able to do that. We would prefer a solution with the drivers baked into the image.

The Ubuntu installation instructions from Azure are straightforward, but they don't work. There's a known issue with the 6.11.x kernels, so the latest images from the marketplace won't work. We have an image based on 6.8.0, but following the instructions upgrades the kernel in the process, so that doesn't work either.

Hence my question: Does anyone have experience with using GPUs in the AzureTRE, and if so, how did you get it to work?

The issue names no repository files, tests, or entry points. Start by reviewing the Azure GPU Driver Extension and Ubuntu NVIDIA driver instructions, then trace how TRE image generalisation and network isolation affect installation. Done should be a documented, reproducible GPU-driver approach for Azure TRE, or a clearly established limitation.