[BUG]: LTO applied to all .cu files

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.11.1

Problem description

The CMake helper pybind11_add_module applies LTO by default (source):

  if(NOT DEFINED CMAKE_INTERPROCEDURAL_OPTIMIZATION)
    if(ARG_THIN_LTO)
      target_link_libraries(${target_name} PRIVATE pybind11::thin_lto)
    else()
      target_link_libraries(${target_name} PRIVATE pybind11::lto)
    endif()
  endif()

Given that .cu files are provided, LTO is applied to them as well and as far as I know CUDA does not yet fully support device code link optimization. The end result is a runtime error stating that the provided module has an undefined symbol: fatbinData (i.e. it is missing the device code completely).

In addition: tested with CUDA 10.2 and CUDA 11.6 out of which both versions experienced this. However, considerable changes to how CMake handles CUDA were made in CMAKE 3.18 and calling pybind11_add_module for CMAKE 3.17 works given that CMAKE_CUDA_ARCHITECTURES changed functionality.

It is not an error in pybind11 per se, but since CUDA does not fully support LTO for device code (as far as I know) it should not be applied as a default for .cu sources.

I am not certain about how pybind11 wants to address this. Regardless, once decide upon an action, I am happy to contribute a PR. Perhaps too much CUDA-specific stuff in pybind11 is not feasible.

Related issues to CUDA and CMAKE are found at these links:
1
2
3
4

@eyalroz, seems to have experienced issues with CMake and CUDA related issues. Would you happen to know about LTO for device code in this setting?

Reproducible example code

Use `pybind11_add_module` instead of `add_library` to build a .so lib.

add_library(${MODULENAME} MODULE ${CPPSOURCES} ${CUSOURCES})
vs
pybind11_add_module(${MODULENAME} ${CPPSOURCES} ${CUSOURCES})

### Is this a regression? Put the last known working version here if it is.

Not a regression

Start in tools/pybind11Tools.cmake around the linked pybind11_add_module LTO logic, then reproduce the difference between add_library and pybind11_add_module with the supplied C++ and .cu sources. Check how CMake 3.18 and later handle CUDA sources and LTO; done should prevent device code from being omitted while preserving the intended LTO behavior for supported sources.