Protobuf version mismatch

Describe the Bug

Python scripts complain about protobuf mismatch between gencode(protoc) and runtime(protobuf) versions.

Steps to Reproduce

docker run -it --rm  -w /tmp ubuntu:22.04 bash

#Within the container
apt -y update; apt -y upgrade
apt -y install coreutils wget vim git make cmake python3.10 python3-pip
pip install --upgrade pip
git clone https://github.com/mlcommons/chakra.git
cd chakra/
pip install .
chakra_generator

#Error message
root@28df32360644:/tmp/chakra# chakra_generator                                                                                                                                                                           
Traceback (most recent call last):                                                                                                                                                                                        
  File "/usr/local/bin/chakra_generator", line 5, in <module>
    from chakra.src.generator.generator import main
  File "/usr/local/lib/python3.10/dist-packages/chakra/src/generator/generator.py", line 3, in <module>
    from ...schema.protobuf.et_def_pb2 import (
  File "/usr/local/lib/python3.10/dist-packages/chakra/schema/protobuf/et_def_pb2.py", line 12, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/usr/local/lib/python3.10/dist-packages/google/protobuf/runtime_version.py", line 106, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/usr/local/lib/python3.10/dist-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected mismatched Protobuf Gencode/Runtime major versions when loading et_def.proto: gencode 6.31.0 runtime 5.29.5. Same major version is required. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Possible Solutions

The two possible fixes are:

1. Edit pyproject.toml to accept protobuf==6.*
2. Find a way to configure setuptools-grpc to accept a specific protoc version (Very much unlikely from my impression of the source code, but still)

Detailed explanation:

Copy & Pasted from https://github.com/astra-sim/astra-sim/pull/314 .

TL, DR: As long as Chakra uses setuptools-grpc in the build process to compile .proto files, regardless of what is written in pyproject.toml, the generated pb2.py file will always be compiled with whatever protoc the grpc repo points to (which is v.6.31.0 as of today)

• gencode 6.31.0 refers to the protoc version used to convert et_def.proto into et_def_pb2.py.
• runtime 5.29.0 refers to the pip protobuf package, that is triggered when running python scripts like python gen_chakra_traces.py.

We know that pip install . installs protobuf v5.29.0 and uses it, as it is the dependency defined in Chakra's pyproject.toml file.
Then where does the 6.31.0 come from?

When running pip install ., Chakra's pyproject.toml declares that it will use setuptools, specifically a plugin called setuptools-grpc whose repo is available (here)[https://github.com/CZ-NIC/setuptools-grpc]. setuptools-grpc imports and uses grpc_tools.protoc. That is, setuptools-grpc does not use the system installed, or even the pip installed protobuf/protoc to compile et_def.proto into et_def_pb2.py. This can be verified by trying to run pip install . without installing protoc in either the system (apt-get -y ...) or pip. You can see that the pb2.py files are created nontheless.

Then what protoc version does setuptools-grpc use? If you go into the grpc repo which holds grpc_tools, you can see that it imports the source code of the protocol buffer as a submodule, and compiles protoc from source. That is, any code calling grpc_tools.protoc is fixed to whichever protoc version is submoduled in the grpc repo.

Looking into the code, we see that currently (2025-JUN-12), the grpc repo points to commit 3d4adad of the protobuf repo. This commit has a tag of v31.0, which corresponds to v6.31.0 in Python. That's where the gencode 6.31.0 comes from.

More notes: Why does v31.0 correspond to v6.31.0 "in Python"?

Ref: https://protobuf.dev/support/version-support/

The protobuf repo has a repo-wide version, which consists only of minor/patch numbers (v31.0, v29.5, etc).
For each repo-wide version, each language may use different major versions. For example, for the repo-wide version v31.0, C++ and Python calls it v6.31.0, while Ruby calls it v4.31.0. This is because the same repo-wide update may have a groundbreaking change in one language, while it has minimal effect in another language. That is, C++ v6.31.0 and Ruby v4.31.0 points to the same commit in the protobuf repo.

So, once we know the repo-wide version v31.0, we look up the table in the link above to find the major number for Python

Start with the Docker reproduction and inspect pyproject.toml, chakra_generator, generator.py, and schema/protobuf/et_def_pb2.py. Compare the protobuf runtime dependency with the version used by setuptools-grpc through grpc_tools.protoc, using the referenced build_grpc.py behavior for context. The work is done when a clean pip install produces matching protobuf major versions and chakra_generator starts without VersionError.