[BUG]: Pybind11 conflicts with TBB because of the `PYBIND11_DEBUG_MARKER` hack

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

b07fddb21993d4fa358822dcbf2b36e0fd20db46

Problem description

Just tracked down a really cryptic bug. Pybind11's PYBIND11_DEBUG_MARKER hack (where it does #undef _DEBUG temporarily, and then defines it back) causes errors in TBB headers.

Here's what happens:

• You set _DEBUG to 1 somehow.

• Then you include any TBB header. They all include tbb/detail/_config.h that defines macro TBB_USE_DEBUG according to the value of _DEBUG:

  • If _DEBUG is undefined, #define TBB_USE_DEBUG 0
  • If _DEBUG is defined to nothing, #define TBB_USE_DEBUG 1
  • Otherwise (_DEBUG == 0 or 1), #define TBB_USE_DEBUG _DEBUG <- this branch is chosen

• Then you #include <pybind11/pybind11.h>. That temporarily #undefines _DEBUG, and then defines it back, but the original value is lost, as it always defines it to empty, regardless of the original value.

• Then you include some other TBB header that does #if TBB_USE_DEBUG. Now TBB_USE_DEBUG expands to empty, and #if with no expression is a compilation error:

  ......./oneapi/tbb/concurrent_vector.h:893:22: error: expected value in expression
  893 |     #if TBB_USE_DEBUG
  

In other words, TBB by itself can handle any value of _DEBUG: both empty and 0/1. What it can't handle is the value suddenly changing from 1 to empty.

Proposed solution:

Improve the PYBIND11_DEBUG_MARKER hack to correctly reproduce the original value of the macro.

Here's how you store the value:

#define __PYBIND11_CONCAT_AUX(A,B) A##B
#define __PYBIND11_IS_MACRO_EMPTY(A,IGNORED) __PYBIND11_CONCAT_AUX(__PYBIND11_MACRO_EMPTY,A)
#define __PYBIND11_MACRO_EMPTY 1

#ifdef _DEBUG
#  if __PYBIND11_IS_MACRO_EMPTY(_DEBUG,IGNORED)==__PYBIND11_MACRO_EMPTY
#    define PYBIND11_DEBUG_MARKER_EMPTY
#  elif _DEBUG
#    define PYBIND11_DEBUG_MARKER_1
#  else
#    define PYBIND11_DEBUG_MARKER_0
#  endif
#  undef _DEBUG
#endif

And this is how you load it back:

#if defined(PYBIND11_DEBUG_MARKER_EMPTY)
#  define _DEBUG
#elif defined(PYBIND11_DEBUG_MARKER_1)
#  define _DEBUG 1
#elif defined(PYBIND11_DEBUG_MARKER_0)
#  define _DEBUG 0
#endif

The macro emptiness check is grabbed straight from TBB.

Reproducible example code

Compile with MSVC (or Clang-cl), and ensure _DEBUG is defined to 1.

#include <tbb/parallel_for.h>
#include <pybind11/pybind11.h>
#include <tbb/enumerable_thread_specific.h>

Is this a regression? Put the last known working version here if it is.

Not a regression

Start in pybind11/pybind11.h, where the PYBIND11_DEBUG_MARKER handling is used, and reproduce the reported include order with TBB while _DEBUG is defined as 1. Verify that the original _DEBUG value is preserved across the pybind11 include and that the TBB headers compile without the #if TBB_USE_DEBUG error.