GPU not found inside long running containers

I've got 4 of these gpu nodes running in a docker compose file:

  odm-node-1:
    <<: *logging_config
    network_mode: "host"
    restart: always
    image: opendronemap/nodeodm:gpu
    container_name: kclf_odm_node_1
    entrypoint:
      - /usr/bin/node
      - /var/www/index.js
      - -p
      - ${DOCKER_EXPOSE_ODM_NODE_PORT}
      - "-q 28"
      - "--max_concurrency 28"
    deploy:
      resources:
        limits:
          cpus: "28"
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]
    volumes:
      - /home/dronefiles/odm-node-1:/var/www/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
  odm-node-2:
    <<: *logging_config
    network_mode: "host"
    restart: always
    image: opendronemap/nodeodm:gpu
    container_name: kclf_odm_node_2
    entrypoint:
      - /usr/bin/node
      - /var/www/index.js
      - -p
      - ${DOCKER_EXPOSE_ODM_NODE2_PORT}
      - "-q 28"
      - "--max_concurrency 28"
    deploy:
      resources:
        limits:
          cpus: "28"
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["1"]
              capabilities: [gpu]
    volumes:
      - /home/dronefiles/odm-node-2:/var/www/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
  odm-node-3:
    <<: *logging_config
    network_mode: "host"
    restart: always
    image: opendronemap/nodeodm:gpu
    container_name: kclf_odm_node_3
    entrypoint:
      - /usr/bin/node
      - /var/www/index.js
      - -p
      - ${DOCKER_EXPOSE_ODM_NODE3_PORT}
      - "-q 28"
      - "--max_concurrency 28"
    deploy:
      resources:
        limits:
          cpus: "28"
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - /home/dronefiles/odm-node-3:/var/www/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
  odm-node-4:
    <<: *logging_config
    network_mode: "host"
    restart: always
    image: opendronemap/nodeodm:gpu
    container_name: kclf_odm_node_4
    entrypoint:
      - /usr/bin/node
      - /var/www/index.js
      - -p
      - ${DOCKER_EXPOSE_ODM_NODE4_PORT}
      - "-q 28"
      - "--max_concurrency 28"
    deploy:
      resources:
        limits:
          cpus: "28"
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["1"]
              capabilities: [gpu]

Every 2 nodes share a single GPU to avoid unnecessary contention.

Sporadically, the containers lose the GPU and when running nvidia-smi error code is something like:

Failed to initialize NVML

I've prepared a monkey patch which runs nvidia-smi inside the container and restarts it if it sees the error string.

Since this is likely an upstream issue, it would probably make sense to check that the GPU is available prior to processing and warn the user, and switch off whatever gpu settings there are. I've seen processing fail in the upstream ODM when GPU isn't available. I dont have that error code on hand atm.

In python, I do the following before creating a task for nodeODM:

  output = invidia_test(container)
  failed_nvidia = 'Failed to initialize NVML' in output
  if failed_nvidia and process_queue == 0:
    container.restart()

Start by tracing the Python path before creating a task for nodeODM, including invidia_test and container.restart, and compare it with the reported nvidia-smi failure inside long-running containers. Determine how GPU availability should be checked before processing, how users should be warned, and what GPU settings should be disabled when NVML initialization fails.