facebookresearch / facebookresearch/IntPhys2

Evaluation code to parse raw losses into metrics

Open
#8 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
112
Forks
14
PR merge metrics
No merged PRs in 30d

Description

Hi -- thanks for the work & code!

If I am not mistaken, the ```eval.py``` code saves a dictionary containing the raw losses as :
```losses_{frame_step}fs_{"_".join([str(ctxt) for ctxt in all_context_lengths])}ctxt_rank{rank}.pth```

However, I do not see any code to read in that file to compute metrics. It seems the provided Jupyter notebook reads in a CSV file from LLM outputs. Could you please share the code used to obtain metrics from JEPA-style .pth files (i.e. raw losses)?

While it is possible to adapt it from Garrido et al.'s ```jepa-intuitive-physics``` repo, sharing the code used here would help reproducibility.

Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.