Combining apex with hyperopt

Hi!

Thank you for this library.

I have a use case where I am using hyperopt for finding optimal hyperparameters for my model. The procedure requires that for every hyperparameter iteration, my model gets reinitialized with new hyperparameters and gets trained for some amount of iterations. Consequently, we must also call amp.initialize(...) on this newly initialized model for every hyperparameter iteration.

When I call amp.initialize(...) for the second time I always get nan values during training. I know from the documentation that you shouldn't call amp more than once, but I really need it in this context.

I am wondering if someone would help me with this?

Thank you!

Best regards,
Robert

Start with the documented amp.initialize(...) entry point and reproduce the second initialization while hyperopt reinitializes and retrains the model. Check whether the NaN values occur only after the first initialization; done means determining whether repeated initialization is supported or documenting the limitation.