MolmoAct-7B-D action head decodes to near-zero / input-independent under the documented recipe (reproduces in HF and vLLM)
- Lingua principale
- Python
- Stelle
- 383
- Fork
- 43
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
## Summary
Running `allenai/MolmoAct-7B-D-0812` with the **documented HF quick-start recipe** — and, as a cross-check, with **your own vLLM path** (`SteerSimplerEnv/molmoact_model_test.py` style, `vllm==0.8.5`) — the model's **visual reasoning is coherent and input-dependent** (it emits a sensible depth map and an image-space end-effector trajectory that changes with the scene), but the **decoded 7-D low-level action collapses to a near-zero, effectively input-independent vector**. I wanted to flag this in case a step in the action decoding is missing on my end, or there's a known checkpoint issue.
## Evidence
**1. The model's own example decodes near-zero.** Using the exact quick-start (its prompt, `example_1.png` + `example_2.png`, "close the box"):
```
parse_action(generated_text, unnorm_key="molmoact")
# -> [-8.6e-06, -6.4e-05, -1.4e-04, -4.0e-04, 1.3e-04, 8.0e-05, 0.996]
```
Every position/rotation dim is ~an order of magnitude **below the action `std`** from `norm_stats["molmoact"]` (`std ≈ [0.0052, 0.0075, 0.0064, 0.0132, 0.0109, 0.0149]`), and far from the sample action shown in the model card (`[0.073, 0.082, ...]`, which itself exceeds the norm-stats `max` of `~0.06`).
**2. The action is nearly input-independent.** Two clearly different scenes (different object layouts, same instruction) decode to the **byte-identical** position delta `[-1e-05, -6e-05, -0.00029, ...]`, even though the predicted image-space trajectories differ (so the model *is* conditioning on the image for the trajectory, just not for the decoded action).
**3. It reproduces in vLLM (your stack), not just HF.** Running the vLLM path (`olmo.vllm.molmoact.MolmoActForActionReasoning`, `vllm==0.8.5`, `SamplingParams(temperature=0)`) on the same scene yields **byte-identical action tokens** to the HF `.generate()` path — so it does not appear to be an HF-serving artifact.
**4. Invariant to dtype and length.** Same result in `bfloat16` and `float32`, and at `max_new_tokens ∈ {256, 512, 900}` (generation completes; not truncation).
## Environment
- `transformers==4.52.3`, `vllm==0.8.5`, `huggingface_hub==0.33.4`
- 1× A100 80GB, CUDA 12.4, `torch 2.4`
- checkpoint: `allenai/MolmoAct-7B-D-0812`, `unnorm_key="molmoact"`
## Question
Is there an action-decoding step (or a different `unnorm_key` / generation config) required to recover usable low-level actions from this checkpoint via the documented recipe? The depth + trajectory outputs look great; only the final 7-D action appears degenerate. Happy to share a minimal repro script.
Thanks for releasing MolmoAct — the visual-reasoning trace is genuinely useful.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.