microsoft / microsoft/Windows-Containers

Exposing individual GPUs to a container using LocationPath erroneously exposes all GPUs

Open
#333 28 comments 1 reaction 2 assignees View on GitHub

@NAWhitehead is already working on this.

Since Sep 26, 2023.

enhancement
Dominant language
PowerShell
Stars
551
Forks
76
PR merge metrics
No merged PRs in 30d

Description

Describe the bug:

When using the vpci-location-path://PATH device string syntax with containerd v1.7.0 to expose an individual GPU to a process-isolated Windows container based on its PCIe location path, the container will see all GPUs that are present on the host system. This contradicts not only the intended behaviour, but also the behaviour observed when exposing other types of devices to containers using their PCIe location path.

Steps to reproduce the behaviour:

  1. Provision a machine with Windows Server 2022 and at least two GPUs.

  2. Ensure the appropriate device drivers are installed for the GPUs.

  3. Install containerd version 1.7.0. For example, using Markus Lippert's containerd Windows installer you would run the following command:

    containerd-installer.exe --containerd-version 1.7.0 --cni-plugin-version 0.3.0
    
  4. Download and extract the binaries for the latest release of the Kubernetes Device Plugins for DirectX.

  5. Pull a test image that will enumerate the DirectX devices that are visible to the container:

    ctr images pull "index.docker.io/tensorworks/example-device-discovery:0.0.1"
    
  6. Run the test-device-discovery-cpp.exe executable directly on the host system. You should see details listed for all GPUs that are present on the host system.

  7. Run a container using the test image, replacing <PATH> with the PCIe location path for one of the GPUs, as reported in the output of the previous step:

    ctr run --rm --device "vpci-location-path://<PATH>" "index.docker.io/tensorworks/example-device-discovery:0.0.1" testing
    
  8. You will see that the container lists details for all of the GPUs present on the host system, instead of just the individual GPU that was exposed to the container.

Expected behaviour:

When enumerating DirectX devices, the container should only see the individual device that was exposed to it.

Configuration:

  • Edition: Windows Server 2022
  • Base Image being used: Windows Server Core (mcr.microsoft.com/windows/servercore:ltsc2022)
  • Container engine: containerd
  • Container Engine version: v1.7.0

Additional context:

When containerd parses the IDType://ID device string syntax, it places the values in the corresponding fields of the WindowsDevice structure from the OCI runtime specification. hcsshim then acts on these fields in hcsoci.parseAssignedDevices(), where it translates the IDType value to a member of the HCS schema DeviceType enum and assigns the value to the Type field of the HCS schema Device structure. In the case of an IDType value of vpci-location-path (which is translated to the DeviceInstance enum value), the code then assigns the ID value to the LocationPath field. As per the documentation for the HCS schema, the purpose of this field is to specify the PCIe location path of an individual device instance that should be exposed to the container.

It is important to note that the LocationPath field functions correctly when exposing individual instances of other device types, such as COM ports. Below is the output of the chgport command when I ran it directly on a Windows Server 2022 host system that had multiple COM ports:

> chgport

AUX = \DosDevices\COM1
COM1 = \Device\Serial0
COM3 = \Device\Serial1
COM4 = \Device\Serial2

When I exposed an individual COM port to a process-isolated Windows container using the vpci-location-path://PATH device string syntax, the chgport command running inside the container only saw the individual COM port that was exposed, as expected:

> ctr run --rm --device "vpci-location-path://ACPI(_SB_)#ACPI(PCI0)#ACPI(ISA_)#ACPI(COM1)" "mcr.microsoft.com/windows/servercore:ltsc2022" testing chgport

AUX = \DosDevices\COM1
COM1 = \Device\Serial0

The behaviour is entirely different when the allocated device is a GPU, and this appears to be a bug in either the HCS or the Windows kernel itself. Although I can only speculate as to the underlying cause, it is notable that the DirectX Graphics Kernel (located at \Device\DxgKrnl in the Object Manager namespace) is mounted into each container that accesses one or more GPUs. My understanding is that this component is ultimately responsible for DirectX device enumeration, so it is possible that there is a bug in the enumeration logic whereby it simply fails to filter devices based on the caller's Silo.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.