carpedm20 / carpedm20/DCGAN-tensorflow

Training good till epoch 20 then goes horribly bad...

Open
#270 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
7.2k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

I had these settings
CelebA
265 input
128 output
25 epochs
--crop --train
and everything was going fine till epoch 20 then things got very badly? Anybody an idea what's happening and if I can redo it from epoch 19?? Waste of amazon EC2 GPU time and money otherwise?

![train_20_2198](https://user-images.githubusercontent.com/6031569/36940753-33d73ae4-1f4a-11e8-9c52-9d6f197f7c78.png)
![train_20_2298](https://user-images.githubusercontent.com/6031569/36940754-3a49d558-1f4a-11e8-95ab-001a38b2ce44.png)
![train_20_2398](https://user-images.githubusercontent.com/6031569/36940756-3fd1044c-1f4a-11e8-92e8-c8711d3173cc.png)
![train_21_1033](https://user-images.githubusercontent.com/6031569/36940757-42feba9c-1f4a-11e8-936f-8488e0b7f700.png)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported CelebA training setup: 265 input, 128 output, 25 epochs, with --crop --train. Compare the generated outputs around epochs 19–21 and inspect how training checkpoints are saved or resumed. Done means the cause of the degradation and a verified way to resume from epoch 19 are identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.