facebookresearch / facebookresearch/WorldPrediction
Missing precomputed caption files for SocraticLLM evaluation (+ question about action captions)
- Dominant language
- Python
- Stars
- 31
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Hi, thank you for your excellent work on the WorldPrediction benchmark!
I have successfully set up the evaluation pipeline with the video datasets and am currently trying to evaluate a fine-tuned language model using the Socratic LLM pipeline.
However, I noticed that the `SocraticLLM` model class requires precomputed visual caption files (containing `state_captions` and `action_captions`). These are referenced via `precomputed_caption_path` in the config files, but the actual JSON files do not seem to be included in the repository.
Could you please share/open-source these precomputed caption files? And, are the `action_captions` simply the `action_label` strings provided in the `WorldPrediction-WM.json` and `WorldPrediction-PP.json` files, or are they separate descriptive texts generated by a VLM?
Any help would be greatly appreciated. Thanks again for your time and contribution!
Contributor guide
Assessment
This issue has not been assessed yet.