Unable to run vLLM models under NVIDIA MPS mode (GPU shared scheduling)

I'm trying to run a simple vLLM model (e.g., facebook/opt-125m) using NVIDIA MPS (Multi-Process Service) on a Kubernetes setup with the GPU Operator and nvidia.com/gpu.shared resource type. However, vLLM fails to start under MPS, while it works fine when MPS is disabled.

Setup:
GPU: NVIDIA A100 40GB (2 GPUs per node)
CUDA: 12.4
Driver: 550.x
vLLM: 0.9.1
vLLM chart: from production-stack

Runtime mode: nvidia.com/gpu.shared (via MPS + GPU Operator)

• ServingEngineSpec snippet:

servingEngineSpec:
  runtimeClassName: ""
  modelSpec:
  - name: "opt125m"
    repository: "vllm/vllm-openai"
    tag: "latest"
    modelURL: "facebook/opt-125m"
    replicaCount: 1
    requestCPU: 6
    requestMemory: "16Gi"
    requestGPU: 1
    requestGPUType: "nvidia.com/gpu"

• MPS config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mps-config
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      mps:
        renameByDefault: false
        failRequestsGreaterThanOne: false
        resources:
          - name: nvidia.com/gpu
            replicas: 2

• When running under MPS, vLLM fails with:

RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable

Notes:

• MPS control daemon is running.
• CUDA_MPS_ACTIVE_THREAD_PERCENTAGE and CUDA_MPS_PINNED_DEVICE_MEM_LIMIT were tried without success.
• vLLM works fine in non-MPS mode with same setup.
• ConfigMap for MPS is loaded properly (replicas: 2)

Questions:

• Is MPS officially supported with vLLM in the production stack?
• Does vLLM require exclusive access to a GPU, even in shared (MPS) mode?
• Any known incompatibilities with MPS mode or recommendations to make it work?

Thank you.

Start with the production-stack vLLM chart configuration, the ServingEngineSpec snippet, and the NVIDIA MPS ConfigMap described in the report. Reproduce the failure with nvidia.com/gpu.shared and compare it with non-MPS mode; done means establishing whether the CUDA error is a supported incompatibility and documenting or addressing the required configuration.