"Platform Error: SageMaker pipe channel timed out." after one hour of training.

Using the Object detection model Resnet-50 with AugmentedManifestFile I get a time out error (see log below filtered on "pipe") after about an hour of training. If I reduce the manifest size I do not get this error. (I'm also getting the warning `[03/19/2020 16:13:34 WARNING 139923161179968] Expected number of batches: 1932, did not match the number of batches processed: 616. This may happen when some images or annotations are invalid and cannot be parsed. Please check the dataset and ensure it follows the format in the documentation.` not sure if it is related.)

- Maximum runtime is `18000`
- Maximum wait time for managed spot training `36000`
- Managed spot training `Enabled`
- Instance type `ml.p2.xlarge`

(Memory and disk are not running out.)

`[03/19/2020 15:59:12 INFO 139923161179968] The channel 'train' is in pipe input mode under /opt/ml/input/data/train.`

`[03/19/2020 15:59:12 INFO 139923161179968] The channel 'validation' is in pipe input mode under /opt/ml/input/data/validation.`

`[03/19/2020 15:59:12 INFO 139923161179968] The channel 'train' is in pipe input mode under /opt/ml/input/data/train.`

`[03/19/2020 15:59:12 INFO 139923161179968] The channel 'validation' is in pipe input mode under /opt/ml/input/data/validation.`

`[15:59:12] /opt/brazil-pkg-cache/packages/AIApplicationsPipeIterators/AIApplicationsPipeIterators-1.0.1081.0/AL2012/generic-flavor/src/data_iter/src/ease_det_image_iter.cpp:41: ImageDetRecordIOParser: pipe:///opt/ml/input/data/train, use 3 threads for decoding..`

`[15:59:13] /opt/brazil-pkg-cache/packages/AIApplicationsPipeIterators/AIApplicationsPipeIterators-1.0.1081.0/AL2012/generic-flavor/src/data_iter/src/ease_det_image_iter.cpp:41: ImageDetRecordIOParser: pipe:///opt/ml/input/data/validation, use 3 threads for decoding..`

`Platform Error: SageMaker pipe channel timed out.`

`[16:55:10] /opt/brazil-pkg-cache/packages/AIApplicationsPipeIterators/AIApplicationsPipeIterators-1.0.1081.0/AL2012/generic-flavor/src/data_iter/src/channel.cpp:124: (Platform Error) FIFO 4 of the SageMaker pipe channel '/opt/ml/input/data/train' timed out.`

`Platform Error: SageMaker pipe channel timed out.`

`[16:55:10] /opt/brazil-pkg-cache/packages/AIApplicationsPipeIterators/AIApplicationsPipeIterators-1.0.1081.0/AL2012/generic-flavor/src/data_iter/src/channel.cpp:124: (Platform Error) FIFO 4 of the SageMaker pipe channel '/opt/ml/input/data/train' timed out.`

Start by reproducing training with the full and reduced manifests, then inspect the SageMaker pipe input logs for the train and validation channels. Check the reported batch-count mismatch against the dataset format in the documentation and determine whether invalid records or pipe processing explain the timeout. Done means a reproducible diagnosis and documented resolution.

Written by the indexing model from the issue text.