`gr00t` and `openvla-oft` both broken on a clean deploy by unpinned mujoco drift (3.10.0 `mj_fullM` signature change) + a tfmd/protobuf mismatch — fixes for both verified end-to-end (SR 1.0)

## Summary

On a clean checkout of `main`, both LIBERO targets I tried fail. Root-causing them from logs revealed **one shared root cause plus one target-specific one**, both upstream dependency drift since the April/May validation dates:

1. **Shared: unpinned mujoco now resolves to 3.10.0**, whose changed `mj_fullM` signature breaks robosuite 1.4.0's legacy call. This kills `gr00t` at the `setup_libero.sh` env sanity check, and kills `openvla-oft` later, at eval time (its setup has no env-creation check, so the same breakage only surfaces ~9 min in, inside `run_libero_eval.py`).
2. **`openvla-oft` only: tensorflow-metadata 1.21.0 vs protobuf 4.25.9** — declared deps resolve cleanly, but tfmd's code imports `google.protobuf.runtime_version`, which doesn't exist in protobuf 4.25 → ImportError at the prismatic import check.

**Both targets now verified end-to-end with the fixes below:**

- `gr00t`: one-line mujoco pin → eval completes, **success_rate 1.0 on both LIBERO kitchen tasks** (5/5, 3/3), matching the README.
- `openvla-oft`: tfmd `--no-deps` downgrade + the same mujoco pin → **full LIBERO-10 eval completes, 50/50 episodes successful** (zero `success=False` artifacts).

| Target | Fails at | Error | Status |
|---|---|---|---|
| `gr00t` | `setup_libero.sh` env sanity | `mj_fullM(): incompatible function arguments` | **fix verified (SR 1.0)** |
| `openvla-oft` | `[3/6]` prismatic import | `cannot import name 'runtime_version'` | **fix verified** |
| `openvla-oft` | eval (~9 min in) | `mj_fullM(): incompatible function arguments` (same as gr00t) | **fix verified (50/50 episodes)** |

## Environment

- Repo `main`, reproduced 2026-07-26/27, region `us-east-1`
- `gr00t`: `g6.12xlarge`; `openvla-oft`: `g6.xlarge` (auto-selected)
- Client: macOS, Python 3.13, Node 24, CDK 2.1115. `doctor` all green for both.

## Target 1: `gr00t` — `mj_fullM` signature mismatch (fix verified)

Crash in `setup_libero.sh` sanity check:

```
mujoco.mj_fullM(self.sim.model._model, mass_matrix, self.sim.data.qM)
TypeError: mj_fullM(): incompatible function arguments. The following argument types are supported:
1. (m: mujoco._structs.MjModel, d: mujoco._structs.MjData, dst: NDArray[float64] writeable c_contiguous) -> None
```

**Root cause (source-verified):** robosuite 1.4.0 (pulled transitively by LIBERO's `requirements.txt`) calls the legacy `mj_fullM(model, dst, qM)`. Per the [MuJoCo changelog](https://mujoco.readthedocs.io/en/stable/changelog.html), the signature changed to `mj_fullM(model, data, dst)` in **3.10.0 (2026-06-22)**; 3.1–3.9 accept the old form. `setup_libero.sh` pins `numpy` but not `mujoco`, so the resolver installs the latest **mujoco 3.10.0** and the old call site breaks. (Not a numpy dtype/contiguity issue — robosuite passes a correct `np.ndarray(shape=(nv,nv), dtype=float64, order="C")`.)

**Fix (verified end-to-end):** pin mujoco to a pre-3.10 release for the LIBERO env. I injected it right after the existing `numpy==1.26.4` pin in `setup_libero.sh`:

```bash
uv pip install "mujoco==3.3.1"
```

After this, `setup_libero.sh` completes (`Env OK`), the ZMQ policy server loads, and the eval runs:

```
Task 1 KITCHEN_SCENE3 (moka pot): success_rate=1.0 (5/5)
Task 2 KITCHEN_SCENE4 (bowl+drawer): success_rate=1.0 (3/3)
```

which matches the README's reported numbers.

## Target 2: `openvla-oft` — protobuf too old for tensorflow-metadata (fix verified)

Crash in the `[3/6]` prismatic import check (`/tmp/oft_import_check_stage1.log`):

```
prismatic → dlimp → tensorflow_datasets → tensorflow_metadata →
from google.protobuf import runtime_version as _runtime_version
ImportError: cannot import name 'runtime_version' from 'google.protobuf'
```

**Root cause:** the resolved env has `protobuf==4.25.9` (tensorflow 2.15.0 constrains `protobuf>=3.20.3,<5.0.0dev`, per its PyPI metadata). `tensorflow-metadata==1.21.0` **declares** `protobuf>=4.21.6,<=6.32` for py<3.11 (also per PyPI), so pip resolves everything cleanly with no conflict — but tensorflow-metadata's **code** imports `google.protobuf.runtime_version`, which does not exist in protobuf 4.25 (hence the ImportError). Its declared floor is lower than what the code actually requires, so the breakage only shows up at import time. And since tf 2.15 caps protobuf below 5, the env can't move forward to a protobuf that has `runtime_version`.

**Fix (verified):** downgrade `tensorflow-metadata` only, keeping protobuf 4.25.9 untouched:

```bash
pip install --no-cache-dir --no-deps "tensorflow-metadata==1.15.0"
```

- `--no-deps` matters: tfmd 1.15.0 *declares* `protobuf<4.21` for py<3.11, but its generated code (protobuf 3.20-era gencode) runs fine on the 4.25.9 runtime (gencode ≤ runtime), so there is no need to let pip downgrade protobuf.
- With this pin (applied after `pip install -e openvla-oft` and re-applied after the LIBERO requirements step), the previously-fatal prismatic import check **passes** and the eval starts.
- Note: my first attempt pinned `protobuf==3.20.3` together with tfmd — that fixed the import but **broke wandb 0.28** (`ImportError: cannot import name 'Imports' from 'wandb.proto.wandb_telemetry_pb2'`), since current wandb no longer supports protobuf 3.x. The `--no-deps` variant avoids that.

**After the import fix, the eval then hits the same mujoco 3.10.0 `mj_fullM` breakage as `gr00t`** (`robosuite/controllers/base_controller.py:156`, ~9 min into `run_libero_eval.py`) — openvla-oft's LIBERO install also leaves mujoco unpinned, and there is no env-creation check during setup, so it only surfaces at eval time. Same fix applies (`pip install "mujoco==3.3.1"` in the conda env after the LIBERO requirements step).

**Verified end-to-end:** with both fixes (tfmd `--no-deps` + `mujoco==3.3.1`), the full LIBERO-10 eval completes. From `eval.log`:

```
>> Total episodes: 50
>> Total successes: 50
>> Overall success rate: 1.0000 (100.0%)
```

One cosmetic note: `summary.txt` reports `success_rate: unknown` for this target — the summary parser doesn't pick up the openvla eval output format.

## Note on validation dates

README lists `gr00t` validated 2026-04-27 and `openvla-oft` 2026-05-04. The other targets (RLDX/RoboCasa/MolmoAct2) carry June validation dates *and* extensive dependency-pin reconciliations in their footnotes. The April/May targets appear not to have been re-pinned as upstream moved.

**Happy to open a PR covering both fixes — the mujoco pin in both targets' LIBERO install steps, plus the `tensorflow-metadata` pin for `openvla-oft` — if you'd like.**

Start with setup_libero.sh and the LIBERO installation steps for both targets, then inspect the openvla-oft import check and run_libero_eval.py. Verify clean deployments, the dependency pins described in the issue, and the reported Env OK, import-check, and end-to-end LIBERO evaluation results; robosuite/controllers/base_controller.py:156 and the referenced logs provide failure points.

Written by the indexing model from the issue text.