"Failed to load plugin WhatIfToolPluginLoader" caused by protobuf compatibility error

The current release of the What-If Tool plugin includes generated Python protos that are incompatible with the 4.x protobuf runtime, leading to this error at startup when the virtual environment only has the 4.x protobuf runtime:

E0210 14:56:01.342828 139767216988224 application.py:125] Failed to load plugin WhatIfToolPluginLoader.load; ignoring it.
Traceback (most recent call last):
  File "/usr/local/google/home/nickfelt/.tf-venvs/tf-nightly-py3/lib/python3.10/site-packages/tensorboard/backend/application.py", line 123, in TensorBoardWSGIApp
    plugin = loader.load(context)
  File "/usr/local/google/home/nickfelt/.tf-venvs/tf-nightly-py3/lib/python3.10/site-packages/tensorboard_plugin_wit/wit_plugin_loader.py", line 57, in load
    from tensorboard_plugin_wit.wit_plugin import WhatIfToolPlugin
  File "/usr/local/google/home/nickfelt/.tf-venvs/tf-nightly-py3/lib/python3.10/site-packages/tensorboard_plugin_wit/wit_plugin.py", line 40, in <module>
    from tensorboard_plugin_wit._utils import common_utils
  File "/usr/local/google/home/nickfelt/.tf-venvs/tf-nightly-py3/lib/python3.10/site-packages/tensorboard_plugin_wit/_utils/common_utils.py", line 17, in <module>
    from tensorboard_plugin_wit._vendor.tensorflow_serving.apis import classification_pb2
  File "/usr/local/google/home/nickfelt/.tf-venvs/tf-nightly-py3/lib/python3.10/site-packages/tensorboard_plugin_wit/_vendor/tensorflow_serving/apis/classification_pb2.py", line 15, in <module>
    from tensorboard_plugin_wit._vendor.tensorflow_serving.apis import input_pb2 as tensorflow__serving_dot_apis_dot_input__pb2
  File "/usr/local/google/home/nickfelt/.tf-venvs/tf-nightly-py3/lib/python3.10/site-packages/tensorboard_plugin_wit/_vendor/tensorflow_serving/apis/input_pb2.py", line 37, in <module>
    _descriptor.FieldDescriptor(
  File "/usr/local/google/home/nickfelt/.tf-venvs/tf-nightly-py3/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
TensorBoard 2.12.0a20230209 at http://xn--260a.mtv.corp.google.com:6006/ (Press CTRL+C to quit)

Note: this doesn't prevent TensorBoard overall from running, it only leads to the What-If Tool plugin being unavailable, but it's a large and scary error message that users are likely to notice.

We didn't notice this previously because TensorBoard itself had generated protos incompatible with the 4.x runtime (issue https://github.com/tensorflow/tensorboard/issues/5703), but that restriction has recently been lifted with #6147 and released in TensorBoard 2.12. For users who get tensorboard as a dependency of tensorflow, this won't yet affect them because TensorFlow also has required the 3.x protobuf runtime, but the soon-to-be-released TF 2.12 require the 4.x runtime, which will likely result in many more users noticing the error and not having a good workaround.

The solution is for the What-If Tool plugin to release a new version with protos generated with protoc 3.19 or higher.

Start with the vendored tensorflow_serving *_pb2.py files named in the traceback and inspect how the What-If Tool plugin packages and releases them. Verify the regenerated protos against the protobuf 4.x runtime and confirm TensorBoard starts without the plugin-load error; done requires a new What-If Tool plugin release.