cupy / cupy/cupy

Runtime compilation should fall back to PTX rather than CUBIN

Open
#7,455 6 comments 1 reaction 1 assignee Claimed by @kmaehashi View on GitHub
cat:enhancement st:needs-discussion
Dominant language
Python
Stars
12.3k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
45

Description

Currently, if the current device is newer than the device supported in that version of CuPy/CUDA, we are falling back to the latest CC supported by NVRTC:

For build-time compilation (CUB/Thrust compiled via `nvcc`), we are adding PTX for the latest compute capability (assuming CCs are [monotonic](https://github.com/cupy/cupy/pull/7100#discussion_r993216766)): https://github.com/cupy/cupy/blob/f2a9d4bf498f80a46b4f2ebbca6734d741c4716a/install/cupy_builder/_compiler.py#L78
For runtime-compilation (kernels compiled via NVRTC), we are using maximum compute capability supported by the NVRTC: https://github.com/cupy/cupy/blob/f2a9d4bf498f80a46b4f2ebbca6734d741c4716a/cupy/cuda/compiler.py#L156

This was intended to use PTX's forward-compatibility feature:

https://docs.nvidia.com/cuda/ampere-compatibility-guide/

> For example, PTX code generated for compute capability 7.x is supported to run on compute capability 7.x or any higher revision (major or minor), including compute capability 8.x.

However, after #5097 CuPy generates CUBIN instead of PTX at runtime, so this fallback stopped working as expected. Runtime compilation should be done in PTX when fallback is needed.

This was reported on the forum: https://groups.google.com/g/cupy/c/wJdMOJa8Bxo

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.