H100 Support

Search before asking

• I have searched the Inference issues and found no similar feature requests.

Description

Currently the version of CUDA we use in our GPU base images doesn't support the sm_90 compute environment which the H100 needs. I think it needs at least CUDA 11.8 and our base image is 11.7 (FROM nvcr.io/nvidia/cuda:11.7.1-cudnn8-runtime-ubuntu22.04).

Sidenote: Latest is 12.3; may be good to try to get all the way up there if possible while maintaining backwards compatibility to pull in bug fixes & performance improvements.

Use case

People with H100s.

Additional

NVIDIA H100 PCIe with CUDA capability sm_90 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70 sm_75 sm_80 sm_86.
If you want to use the NVIDIA H100 PCIe GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

Locate the GPU base image declaration using nvcr.io/nvidia/cuda:11.7.1-cudnn8-runtime-ubuntu22.04 and inspect how the PyTorch installation is built. Try a CUDA version that supports the H100's sm_90 capability, then verify H100 compatibility while preserving support for existing GPUs.