allenai / allenai/vla-evaluation-harness
BEHAVIOR Challenge 2026: integrate the official evaluator through a protocol bridge
- Dominant language
- Python
- Stars
- 601
- Forks
- 49
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 7
Description
Follow-up to #113, whose immediate request (label the current integration as BEHAVIOR 2025) is handled in the linked PR. This issue tracks the actual 2026 evaluation path.
## Problem
The official 2026 BEHAVIOR Challenge pins BEHAVIOR-1K v3.9.1, evaluates 100 tasks, and drives the policy server through `python -m omnigibson.eval.eval`: the evaluator waits for `/healthz`, sends flattened observations over WebSocket, and expects a msgpack response containing an `action` array. Official RGB + depth evaluation uses `omnigibson.eval.wrappers.RGBDFullResWrapper`, and reported results use instance indices 0-9. Instructions: https://behavior.stanford.edu/challenge/evaluation.html
Porting the harness's `Behavior1KBenchmark` to v3.9.1 would not establish 2026 submission compatibility, because the harness would still re-implement the evaluator and its own episode loop.
## Proposed approach: protocol bridge
Keep the official evaluator as the driver and bridge its wire protocol to the harness model-server protocol:
- New `docker/Dockerfile.behavior1k2026` pinned to BEHAVIOR-1K v3.9.1 and the Isaac Sim version it requires, separate from the existing 2025 image.
- A thin shim server that exposes the official policy-server interface (`/healthz`, flattened observations in, msgpack `{action: [...]}` out) and translates each request to the vla-eval model-server WebSocket protocol.
- The benchmark entry runs `python -m omnigibson.eval.eval` (100 tasks, instances 0-9) inside the container and collects results from the evaluator's output.
This validates the real submission path end to end and reuses every existing model server unchanged. A passing run is then a true 2026 compatibility claim, which the 2025 adapter cannot provide.
## Scope notes
- The 2025 integration stays as is, now explicitly labeled.
- #54 (BEHAVIOR-1K Docker image support) overlaps with the image work here.
Contributor guide
Assessment
This issue has not been assessed yet.