GuyTevet / GuyTevet/motion-diffusion-model

nan value & gt gt2 & KID problems when eval_unconstrained_humanact12

Open
#191 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4.1k
Forks
462
PR merge metrics
No merged PRs in 30d

Description

Hi!

When I ran the eval part, I met a few problems about unconstrained, and hope you can help me.

The process is carried out, and no error is reported to stop the program.

The first problem is that there is no **_KID_** value, which can refer to the screenshot from the `evaluation_results_iter450000_samp1000_scale1_a2m.yaml` file that is the evaluated result for the best model.
![image](https://github.com/GuyTevet/motion-diffusion-model/assets/58409065/23d41343-1caa-447c-bbfc-21b5f66b331c)

In the `.yaml` file, there are many nan values, including **accuracy_gen**, **accuracy_gt**, **accuracy_gt2**, **fid_unconstrained**, **kid_unconstrained**, **multimodality_gen**, **multimodality_gt**, **multimodality_gt2**, **precision_unconstrained**, and **recall_unconstrained**.
![image](https://github.com/GuyTevet/motion-diffusion-model/assets/58409065/2ec74c11-c7d2-43a1-adad-ca2c9ff3c937)
![image](https://github.com/GuyTevet/motion-diffusion-model/assets/58409065/8dab2e74-b8ab-4142-ba74-861c38d48f96)
![image](https://github.com/GuyTevet/motion-diffusion-model/assets/58409065/d9dc8bfd-ff46-49a3-ad03-e92cedfa2f9f)

To double-check the problem, I also evaluated the best model given by the official zip, i.e. `./save/unconstrained/model000450000.pt`
But the problems still exist.
![Screen Shot 2024-02-19 at 18 11 17](https://github.com/GuyTevet/motion-diffusion-model/assets/58409065/cb1d8729-950f-4c98-8171-1653a8e6fc5d)

Finally, I want to know why we need **gt and gt2**, which means two ground truths.

Thanks in advance, and looking forward to your early reply~

`evaluation_results_iter450000_samp1000_scale1_a2m.yaml` file attached here
```
feats:
accuracy_gen:
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
accuracy_gt:
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
accuracy_gt2:
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
diversity_gen:
- '6.89782'
- '6.89011'
- '6.73952'
- '6.74987'
- '6.82585'
- '6.6621'
- '6.82288'
- '6.75882'
- '6.77904'
- '6.92874'
- '6.8772'
- '6.67161'
- '6.83096'
- '6.77628'
- '6.77839'
- '6.62971'
- '7.01509'
- '6.90686'
- '6.81357'
- '6.58429'
diversity_gen_unconstrained: 17.008365631103516
diversity_gt:
- '6.61153'
- '7.01676'
- '6.99224'
- '6.80892'
- '6.91731'
- '6.85943'
- '6.88625'
- '6.94846'
- '7.07225'
- '6.86249'
- '6.76709'
- '7.02609'
- '6.87979'
- '6.87578'
- '6.56889'
- '6.93818'
- '6.68828'
- '6.88319'
- '6.92963'
- '6.8524'
diversity_gt2:
- '6.74916'
- '6.95423'
- '6.71648'
- '6.77055'
- '6.91332'
- '6.54734'
- '6.62869'
- '6.84382'
- '6.66525'
- '6.68556'
- '6.70404'
- '6.99287'
- '6.78967'
- '6.69709'
- '6.98212'
- '6.87764'
- '6.74566'
- '6.93538'
- '6.85223'
- '6.6096'
diversity_gt_unconstrained: 20.708080291748047
fid_gen:
- '0.297286'
- '0.259686'
- '0.422459'
- '0.228987'
- '0.320257'
- '0.30219'
- '0.284305'
- '0.314953'
- '0.262808'
- '0.298342'
- '0.217695'
- '0.361196'
- '0.412756'
- '0.210205'
- '0.232225'
- '0.309696'
- '0.294808'
- '0.299409'
- '0.277911'
- '0.237049'
fid_gt:
- '-2.84217e-14'
- '-7.10543e-15'
- '3.55271e-14'
- '-6.39488e-14'
- '-7.81597e-14'
- '-3.55271e-14'
- '-9.9476e-14'
- '5.68434e-14'
- '-2.13163e-14'
- '-4.9738e-14'
- '7.10543e-15'
- '-2.84217e-14'
- '-2.13163e-14'
- '-2.13163e-14'
- '2.84217e-14'
- '-7.81597e-14'
- '-9.9476e-14'
- '-7.81597e-14'
- '-9.23706e-14'
- '7.10543e-15'
fid_gt2:
- '0.0515395'
- '0.0658431'
- '0.0506913'
- '0.0591474'
- '0.0492066'
- '0.0688688'
- '0.0487507'
- '0.0502104'
- '0.0412811'
- '0.0510524'
- '0.0544728'
- '0.060396'
- '0.0428596'
- '0.0442645'
- '0.0322802'
- '0.04161'
- '0.0486726'
- '0.0648593'
- '0.0743409'
- '0.0594989'
fid_unconstrained: !!python/object/apply:numpy.core.multiarray.scalar
- &id001 !!python/object/apply:numpy.dtype
args:
- f8
- false
- true
state: !!python/tuple
- 3
- <
- null
- null
- null
- -1
- -1
- 0
- !!binary |
iLdXr9vzQEA=
kid_unconstrained: !!python/object/apply:numpy.core.multiarray.scalar
- *id001
- !!binary |
FIX+FTG/2T8=
multimodality_gen:
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
multimodality_gt:
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
multimodality_gt2:
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
precision_unconstrained: null
recall_unconstrained: null
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the unconstrained evaluation with save/unconstrained/model000450000.pt and compare the generated evaluation_results_iter450000_samp1000_scale1_a2m.yaml with the reported values. Trace the evaluation entry point and metric calculations for accuracy, multimodality, KID, precision, and recall, including how gt and gt2 are supplied. Done means the cause of the NaN and missing KID values is identified and the gt/gt2 usage is clarified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.