genrules with `py_binary` in `exec_tools` broken by TF python toolchain

The problem:

1. We have genrules that use py_binary targets listed in exec_tools.
   • Motivating case: //tensorboard/plugins/debugger_v2/tf_debugger_v2_plugin:generate_dtypes
2. Because they are listed in exec_tools these py_binary targets get built in exec mode, using a special exec build configuration (rather than the host mode, which is what happens if you pass them in tools, and rather than the target mode, which is what happens if you just build the py_binary directly).
   • Quoting @wchargin: "host" = the machine on which Bazel runs, "execution" = the machine on which Bazel actions execute, "target" = end-user architecture
   • See background note at bottom.
3. Due to our use of the TF Bazel workspace, we inherit a bunch of complex configs that affect the exec build configuration, and in particular, we inherit a special TF python toolchain that overrides the default Bazel autodetecting python toolchain.
4. The special TF python toolchain finds the full path to the python binary on the build machine, and then "bakes" this into the Python toolchain configuration, which results in the exec-mode build of the py_binary rule's wrapper script hardcoding the path to whatever python was in use when it was first built.
5. This build artifact has an environment dependency - most obviously on PATH - but unlike other build artifacts that rebuild whenever PATH changes (https://github.com/tensorflow/tensorboard/issues/3213), this build artifact doesn't get marked as stale if PATH changes (possibly because it's assumed to be an exec mode artifact and thus "independent" of the host machine, even though it's certainly not), and consequently seems to persist until you do bazel clean --expunge or possibly some other scenarios (in practice it seemed to occasionally get cleared for no apparent reason at all).
6. When PATH changes - for example, switching to a new virtualenv - everything else in our build will reflect the new python interpreter path (since it uses the Bazel auto-detecting python toolchain), except for these py_binary targets, which continue to be hardcoded to point to the old path.
7. At this point, not only is the build not hermetic, it's invisibly dependent on the prior build environment, and as soon as something breaks there (for example: numpy gets uninstalled) you have a breakage that is not reliably reproducible and generally maddening to debug.
8. (╯°□°)╯︵ ┻━┻

Background on exec_tools: Bazel genrules have historically always run in host mode, and as such, the tools attribute to genrules always builds its binaries in host mode. This was a problem for python 2 -> 3 since Bazel's host mode can only run either python 2 or python 3, but can't do both. So genrules had an exec_tools attribute introduced in https://github.com/bazelbuild/bazel/issues/6443 to allow tool deps for genrules to be build in exec mode and vary whether they used python 2 or python 3 on a per-target basis. In preparation for switching over the Bazel host mode default from 2 to 3, we were asked to migrate all our genrules from tools to exec_tools, which we did in https://github.com/tensorflow/tensorboard/pull/4031. Now that they've cut over, nothing really prevents us from switching exec_tools back to tools to simplify our lives, except the note from Bazel team here.

How do we actually inherit the TF toolchain?

• https://github.com/tensorflow/tensorboard/blob/1501f462ad61eba0a8b93910d3995e50fbaab784/WORKSPACE#L83-L97
• https://cs.opensource.google/tensorflow/tensorflow/+/refs/tags/v2.3.0:tensorflow/workspace.bzl;l=76;drc=88e3eebad5c31b69df9dffe4ef244241e2cda86d
• https://cs.opensource.google/tensorflow/tensorflow/+/refs/tags/v2.3.0:tensorflow/workspace.bzl;l=93;drc=88e3eebad5c31b69df9dffe4ef244241e2cda86d
• https://cs.opensource.google/tensorflow/tensorflow/+/refs/tags/v2.3.0:third_party/toolchains/remote_config/configs.bzl;l=6;drc=18569dde74a866758b2008042a6ed2e3c8bdb027
• https://cs.opensource.google/tensorflow/tensorflow/+/refs/tags/v2.3.0:third_party/toolchains/remote_config/rbe_config.bzl;l=152;drc=10c005554d9be2275be7579f00d572308c4dc9a9
• https://cs.opensource.google/tensorflow/tensorflow/+/refs/tags/v2.3.0:third_party/toolchains/remote_config/rbe_config.bzl;l=158;drc=10c005554d9be2275be7579f00d572308c4dc9a9
• https://cs.opensource.google/tensorflow/tensorflow/+/refs/tags/v2.3.0:third_party/py/python_configure.bzl;l=275;drc=9ca9a4f8e9b4d60408a092db4320805711367042
• local_python_configure is more-or-less the root of the issue; it's a repository_rule that bakes many attributes of the local python environment into a synthetic repository using this BUILD file template, including the py_toolchain rule that's causing trouble for us.
• It might be possible for this repository_rule to pass different arguments that make it less of a footgun: https://docs.bazel.build/versions/master/skylark/lib/globals.html#repository_rule
  • In particular, if it included PATH in its environ argument, Bazel claims "If an environment variable in that list change, the repository will be refetched." And it does indeed use PATH to resolve the python binary path it bakes into its python toolchain, as made clear by the get_python_bin() helper error message.
  • Since this is a local repo, in theory it could also set local = True to be regenerated on every fetch, though this repo rule does enough setup that it's probably a non-trivial amount of time to rerun it on every single build.

Start with WORKSPACE lines 83–97 and the generate_dtypes target in tensorboard/plugins/debugger_v2/tf_debugger_v2_plugin/BUILD, then trace the exec_tools behavior and TensorFlow's local_python_configure through the linked toolchain files. Reproduce the stale interpreter-path behavior across PATH or virtualenv changes; done should prevent exec-mode py_binary artifacts from retaining an obsolete Python path.