No Val/Test loss for each epoch?

Hi there,

After a few days experiments with DeepAR, I’m surprised about how powerful DeepAR it for the multivariate time series forecasting problems.

But I do have a question about the role testing data channel, I initially thought it would serve the purpose as a validation set, but it seems there is no validation loss reported after each epoch, and it won’t allow me to customize metrics as it’s one of the built AWS algorithms. Instead, it only report the test RMSE and test loss after all the epochs.

I tried to read through all the docs and original paper, but still didn’t find any clue on this.

Without the validation loss curve, how would I know if my model is overfitting or not during the training process and stop it early if it does?

Thanks in advance for any suggestions.

Start with the DeepAR documentation, the original paper, and the training and testing data-channel behavior described in the issue. Determine whether validation loss, custom metrics, or early stopping are supported, then document how a user can identify overfitting and what output or workflow constitutes a complete answer.

Written by the indexing model from the issue text.