[BUG] 2.11.1: `pybind11` cmake module broken with `-D USE_PYTHON_INCLUDE_DIR=OFF`

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.11.1

Problem description

Looks like when pybind11is build with cmake -D USE_PYTHON_INCLUDE_DIR=OFF generated pybind11 cmake modules still expects to find header files in python siterlib tree.
cmake output:

----------------------------- Captured stdout call -----------------------------
*** scikit-build-core 0.5.1 using CMake 3.27.6 (wheel)
*** Configuring CMake...
loading initial cache file /tmp/tmpcr9l27qj/build/CMakeInit.txt
-- The CXX compiler identification is GNU 13.2.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python: /usr/bin/python3 (found version "3.8.18") found components: Interpreter Development.Module
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /usr/lib/python3.8/site-packages/pybind11/include (found version "2.11.1")
-- Configuring done (1.2s)
-- Generating done (0.0s)
----------------------------- Captured stderr call -----------------------------
CMake Error in CMakeLists.txt:
  Imported target "pybind11::headers" includes non-existent path

    "/usr/lib/python3.8/site-packages/pybind11/include"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.



CMake Generate step failed.  Build files cannot be regenerated correctly.

Reproducible example code

I found that issue on testing scikit-build-core
https://github.com/scikit-build/scikit-build-core/issues/368#issuecomment-1742178319

Is this a regression? Put the last known working version here if it is.

Not a regression

Reproduce the failure by configuring pybind11 with -D USE_PYTHON_INCLUDE_DIR=OFF and inspect the generated CMake modules and pybind11::headers target. Verify how the Python site-packages include path is selected. Done means CMake generation succeeds without referencing the nonexistent pybind11/include path.