Use multiple GPUs to process queue

I am trying to use both of my GPUs who are passed through to my docker container.

services: faster-whisper-server-cuda: image: fedirz/faster-whisper-server:latest-cuda build: dockerfile: Dockerfile.cuda context: . platforms: - linux/amd64 - linux/arm64 restart: unless-stopped ports: - 8162:8000 environment: - WHISPER__MODEL=deepdml/faster-whisper-large-v3-turbo-ct2 - WHISPER__INFERENCE_DEVICE=cuda - WHISPER__COMPUTE_TYPE=int8 - WHISPER__NUM_WORKERS=4 - WHISPER__CPU_THREADS=4 - WHISPER_DEVICE=cuda - DEFAULT_LANGUAGE=en - PRELOAD_MODELS=["deepdml/faster-whisper-large-v3-turbo-ct2"] volumes: - hugging_face_cache:/root/.cache/huggingface privileged: true deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] volumes: hugging_face_cache:
I tried everything but it won't use more than 1 GPU even if:

Start with the Docker Compose service definition and Dockerfile.cuda, then inspect how WHISPER__NUM_WORKERS, WHISPER__INFERENCE_DEVICE, and the NVIDIA device reservation are consumed. Reproduce the configuration with both GPUs passed through and determine the project entry point responsible for queue processing. Done means the service demonstrably distributes queued work across both GPUs.