facebookresearch / facebookresearch/diffh2o
Reproduce value from paper
- Dominant language
- Python
- Stars
- 45
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
Hi, thank you for your excellent work!
I'm trying to reproduce the values reported in Table 2 of your paper.
Using the pretrained model and the following command:
```
python -m sample.generate_2stage \
--model_path ./save/diffh2o_full_detailed/model000200000.pt \
--num_samples 16 \
--guidance \
--text_detailed \
--eval_grasp_reference \
--eval_entire_set
```
I evaluated the entire dataset and obtained the following results:
```
iv: 3.6832932692307696
id: 4.474283497497306
id_max: 12.220349985779732
cr: 0.05996021163058345
jp: 0.0012218059557710863
ja: 0.06917452691116986
nc: nan
od: 0.15753278368781132
t_vels: 0.007743637077510357
acc_glob_pos_mean: 0.019779082387685776
acc_loc_pos_mean: 0.006034958176314831
acc_glob_rot_mean: 0.09586863964796066
acc_loc_rot_mean: 0.03712637722492218
handedness_accuracy: 0.9230769230769231
grasp_reference_error: 0.33896538615226746
```
However, several metrics show significant differences compared to Table 2:
| | My Result | Table 2 |
| :-- | :-- | :-- |
| iv | 3.68 | 7.40 |
| ge | 0.34 | 0.12 |
| ha | 0.92 | 0.87 |
| tel | 0.007 | 0.23 |
Did I miss something in the evaluation setup?
Any hints about additional configuration, dataset splits, or evaluation details would be greatly appreciated.
Thanks again for sharing this great work!
Contributor guide
Assessment
This issue has not been assessed yet.