google-deepmind / google-deepmind/deepmind-research
Questions on reproducing ScratchGAN results
- Dominant language
- Jupyter Notebook
- Stars
- 15.2k
- Forks
- 2.9k
- PR merge metrics
- No merged PRs in 30d
Description
Hi, thank you for keeping the code open-sourced! I am wondering if there is something I missed to reproduce the results of scratchGAN on EMNLP News dataset. The optimal FED is **0.0214** whereas that in the original paper was **0.015**.
The results on single NVIDIA Titan RTX are summarized as follows:
| checkpoint | mean train prob | mean valid prob | mean gen prob | FED |
| -------------- | ---------------------- | --------------------- | ---------------------| ---------------- |
| scratchgan-392002 | 0.0234371495898813 | 0.021976527175866067 | 0.016274933121167123 | 0.02389448 |
| scratchgan-400000 | 0.022109536337666214 | 0.021964834304526448 | 0.015994238085113466 | 0.02222306 |
| scratchgan-394002 | 0.023769843741320074 | 0.022377881105057895 | 0.016788717941381037 | 0.023813704 |
| scratchgan-398002 | 0.025539066176861525 | 0.024043583893217146 | 0.018643759249243885 | 0.02179827 |
| scratchgan-396002 | 0.024058140232227743 | 0.022415030980482697 | 0.016733897908125073 | **0.021409184** |
The hyper-parameter settings are default in the source code and the same as those reported in Appendix D.2. The `num_steps` was set to 200,000 by default.
Looking forward to your reply;)
Contributor guide
Assessment
This issue has not been assessed yet.