GuyTevet / GuyTevet/motion-diffusion-model
nan value & gt gt2 & KID problems when eval_unconstrained_humanact12
- Dominant language
- Python
- Stars
- 4.1k
- Forks
- 462
- PR merge metrics
- No merged PRs in 30d
Description
Hi!
When I ran the eval part, I met a few problems about unconstrained, and hope you can help me.
The process is carried out, and no error is reported to stop the program.
The first problem is that there is no **_KID_** value, which can refer to the screenshot from the `evaluation_results_iter450000_samp1000_scale1_a2m.yaml` file that is the evaluated result for the best model.

In the `.yaml` file, there are many nan values, including **accuracy_gen**, **accuracy_gt**, **accuracy_gt2**, **fid_unconstrained**, **kid_unconstrained**, **multimodality_gen**, **multimodality_gt**, **multimodality_gt2**, **precision_unconstrained**, and **recall_unconstrained**.



To double-check the problem, I also evaluated the best model given by the official zip, i.e. `./save/unconstrained/model000450000.pt`
But the problems still exist.

Finally, I want to know why we need **gt and gt2**, which means two ground truths.
Thanks in advance, and looking forward to your early reply~
`evaluation_results_iter450000_samp1000_scale1_a2m.yaml` file attached here
```
feats:
accuracy_gen:
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
accuracy_gt:
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
accuracy_gt2:
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
diversity_gen:
- '6.89782'
- '6.89011'
- '6.73952'
- '6.74987'
- '6.82585'
- '6.6621'
- '6.82288'
- '6.75882'
- '6.77904'
- '6.92874'
- '6.8772'
- '6.67161'
- '6.83096'
- '6.77628'
- '6.77839'
- '6.62971'
- '7.01509'
- '6.90686'
- '6.81357'
- '6.58429'
diversity_gen_unconstrained: 17.008365631103516
diversity_gt:
- '6.61153'
- '7.01676'
- '6.99224'
- '6.80892'
- '6.91731'
- '6.85943'
- '6.88625'
- '6.94846'
- '7.07225'
- '6.86249'
- '6.76709'
- '7.02609'
- '6.87979'
- '6.87578'
- '6.56889'
- '6.93818'
- '6.68828'
- '6.88319'
- '6.92963'
- '6.8524'
diversity_gt2:
- '6.74916'
- '6.95423'
- '6.71648'
- '6.77055'
- '6.91332'
- '6.54734'
- '6.62869'
- '6.84382'
- '6.66525'
- '6.68556'
- '6.70404'
- '6.99287'
- '6.78967'
- '6.69709'
- '6.98212'
- '6.87764'
- '6.74566'
- '6.93538'
- '6.85223'
- '6.6096'
diversity_gt_unconstrained: 20.708080291748047
fid_gen:
- '0.297286'
- '0.259686'
- '0.422459'
- '0.228987'
- '0.320257'
- '0.30219'
- '0.284305'
- '0.314953'
- '0.262808'
- '0.298342'
- '0.217695'
- '0.361196'
- '0.412756'
- '0.210205'
- '0.232225'
- '0.309696'
- '0.294808'
- '0.299409'
- '0.277911'
- '0.237049'
fid_gt:
- '-2.84217e-14'
- '-7.10543e-15'
- '3.55271e-14'
- '-6.39488e-14'
- '-7.81597e-14'
- '-3.55271e-14'
- '-9.9476e-14'
- '5.68434e-14'
- '-2.13163e-14'
- '-4.9738e-14'
- '7.10543e-15'
- '-2.84217e-14'
- '-2.13163e-14'
- '-2.13163e-14'
- '2.84217e-14'
- '-7.81597e-14'
- '-9.9476e-14'
- '-7.81597e-14'
- '-9.23706e-14'
- '7.10543e-15'
fid_gt2:
- '0.0515395'
- '0.0658431'
- '0.0506913'
- '0.0591474'
- '0.0492066'
- '0.0688688'
- '0.0487507'
- '0.0502104'
- '0.0412811'
- '0.0510524'
- '0.0544728'
- '0.060396'
- '0.0428596'
- '0.0442645'
- '0.0322802'
- '0.04161'
- '0.0486726'
- '0.0648593'
- '0.0743409'
- '0.0594989'
fid_unconstrained: !!python/object/apply:numpy.core.multiarray.scalar
- &id001 !!python/object/apply:numpy.dtype
args:
- f8
- false
- true
state: !!python/tuple
- 3
- <
- null
- null
- null
- -1
- -1
- 0
- !!binary |
iLdXr9vzQEA=
kid_unconstrained: !!python/object/apply:numpy.core.multiarray.scalar
- *id001
- !!binary |
FIX+FTG/2T8=
multimodality_gen:
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
multimodality_gt:
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
multimodality_gt2:
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
- nan
precision_unconstrained: null
recall_unconstrained: null
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the unconstrained evaluation with save/unconstrained/model000450000.pt and compare the generated evaluation_results_iter450000_samp1000_scale1_a2m.yaml with the reported values. Trace the evaluation entry point and metric calculations for accuracy, multimodality, KID, precision, and recall, including how gt and gt2 are supplied. Done means the cause of the NaN and missing KID values is identified and the gt/gt2 usage is clarified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100