facebookresearch / facebookresearch/WorldPrediction

Missing precomputed caption files for SocraticLLM evaluation (+ question about action captions)

Open
#5 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
31
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Hi, thank you for your excellent work on the WorldPrediction benchmark!

I have successfully set up the evaluation pipeline with the video datasets and am currently trying to evaluate a fine-tuned language model using the Socratic LLM pipeline.

However, I noticed that the `SocraticLLM` model class requires precomputed visual caption files (containing `state_captions` and `action_captions`). These are referenced via `precomputed_caption_path` in the config files, but the actual JSON files do not seem to be included in the repository.

Could you please share/open-source these precomputed caption files? And, are the `action_captions` simply the `action_label` strings provided in the `WorldPrediction-WM.json` and `WorldPrediction-PP.json` files, or are they separate descriptive texts generated by a VLM?

Any help would be greatly appreciated. Thanks again for your time and contribution!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.