fix(python): PEP 517 builds from sdist produce wheels with missing proto stubs

Agent Diagnostic

• No OpenShell-specific skill applies (this is a build-system packaging issue, not a runtime bug)
• Investigated pyproject.toml build-system configuration, .gitignore rules, mise task graph, Dockerfile.python-wheels, and upstream CI workflows
• Built sdist with maturin sdist and confirmed _proto/ contains only __init__.py
• Built wheel from git checkout with maturin 1.7.8 and 1.11.5; both include proto stubs (because stubs were pre-generated on disk)
• Confirmed [build-system].requires lists only maturin, not grpcio-tools
• Confirmed no maturin build hook or custom build script invokes protoc
• Root cause: proto generation is wired only through the mise task graph (python:proto dependency on build:python:wheel:linux), not through PEP 517 build metadata
• Searched existing issues: #1705 (closed by #2029) addressed gitignore-vs-include, but not the missing generation step
• Tested on current main (v0.0.96)

Description

The Python wheel ships protobuf-generated stubs (*_pb2.py, *_pb2_grpc.py, *.pyi) under openshell/_proto/. These stubs are:

1. Generated by mise run python:proto (invokes grpc_tools.protoc)
2. Gitignored (.gitignore lines 100-101: python/openshell/_proto/* with !__init__.py)
3. Listed in [tool.maturin].include globs so maturin force-includes them despite gitignore

This works when building from a git checkout where python:proto has already run (all upstream CI paths). It breaks when building from the source tarball (sdist), because:

• The sdist does not contain the generated stubs (they are gitignored and not committed)
• [build-system].requires lists only maturin>=1.5,<2.0, not grpcio-tools
• No maturin build hook or build script triggers proto generation
• The include globs silently match nothing
• The resulting wheel has _proto/__init__.py only

__init__.py does from . import datamodel_pb2, openshell_pb2, so importing the package hard-fails with ImportError.

This affects any PEP 517 build frontend (pip, build, fromager) building from the sdist, and any downstream rebuild pipeline that does not replicate the mise task graph.

Reproduction Steps

1. Generate sdist:

   maturin sdist --out /tmp/sdist-test
   

2. Inspect sdist contents:

   tar tzf /tmp/sdist-test/openshell-*.tar.gz | grep _proto
   # Only __init__.py present
   

3. Extract and build wheel from sdist:

   mkdir /tmp/sdist-repro && cd /tmp/sdist-repro
   tar xzf /tmp/sdist-test/openshell-*.tar.gz
   cd openshell-*/
   maturin build --release --out /tmp/wheels-from-sdist
   

4. Inspect wheel:

   unzip -l /tmp/wheels-from-sdist/*.whl | grep _proto
   # Only __init__.py — no *_pb2.py files
   

5. Verify import fails:

   pip install /tmp/wheels-from-sdist/*.whl
   python -c "import openshell._proto"
   # ImportError: cannot import name 'datamodel_pb2'
   

Environment

• OS: Fedora 44 (kernel 7.1.3)
• maturin: tested 1.7.8 and 1.11.5
• OpenShell: v0.0.96 (current main)
• Latest release checked: yes, issue present on main
• Possible duplicates checked: #1705 (closed by #2029) addressed include paths but not the missing generation step

Possible Fixes

Option A: Add grpcio-tools to [build-system].requires and wire a build script that runs protoc before maturin collects files.

Option B: Stop gitignoring the generated stubs and commit them. They would then ship in the sdist and be found by the include globs. Requires regeneration discipline on proto changes.

Option C: Generate stubs in a maturin pre-build hook (if maturin supports one).

Start with pyproject.toml build-system and maturin settings, then inspect the mise python:proto task and the .gitignore rules for python/openshell/_proto. Compare the sdist and wheel contents using the reproduction commands, and verify that a wheel built from the sdist can import openshell._proto with all generated stubs present.