Retraining from scratch yields worse results

Hello,

As written in the paper (2.2 _Deriving Architectures_) I tried to re-retrain from scratch the best derived model, but it surprisingly gives worse result when I retrain it from scratch than if I would keep the original (shared) weights.
I expected training the best model (dag) from scratch to be faster and eventually have a better perplexity, but it's not the case.

I do the following:

1. Launch ENAS with the `--load_path` argument, which loads a previous run, and the `--mode test`, which will call a custom `test` method inside the `trainer` class
2. (In the `test` method) I reset the shared weights with `self.shared.reset_parameters()`
3. I derive the best model (dag)
4. Then I train this model from scratch, iterating over the train set for N epochs (like in the `train_shared` method)

The following picture shows the loss and ppl during the "normal" training (first slope) and after reseting the shared weights (second slope). The second slope only trains the same best model (dag).


Has anyone any idea about why resetting the shared weight and re-training from scratch is so bad?

Start by reviewing the custom test method, self.shared.reset_parameters(), the model derivation step, and the train_shared method described in the issue. Compare the normal shared-weight training path with the from-scratch path and inspect the reported loss and perplexity behavior. Done means identifying a reproducible cause or a clearly documented explanation for the worse retraining results.

Written by the indexing model from the issue text.