sagemaker.exceptions.UnexpectedStatusException: Error for Training job

2022-06-02 17:28:14 Failed - Training job failed
ProfilerReport-1654189943: Stopping
Traceback (most recent call last):
File "code/EamesJointEmbedding/src/eames_joint_embedding/train/trigger_train.py", line 89, in
launch_training(config, wait)
File "code/EamesJointEmbedding/src/eames_joint_embedding/train/trigger_train.py", line 68, in launch_training
estimator.fit(config["inputs"], wait=wait)
File "/opt/conda/lib/python3.6/site-packages/sagemaker/estimator.py", line 682, in fit
self.latest_training_job.wait(logs=logs)
File "/opt/conda/lib/python3.6/site-packages/sagemaker/estimator.py", line 1625, in wait
self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)
File "/opt/conda/lib/python3.6/site-packages/sagemaker/session.py", line 3681, in logs_for_job
self._check_job_status(job_name, description, "TrainingJobStatus")
File "/opt/conda/lib/python3.6/site-packages/sagemaker/session.py", line 3245, in _check_job_status
actual_status=status,
sagemaker.exceptions.UnexpectedStatusException: Error for Training job cluster-eames-: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
Command "/opt/conda/bin/python3.6 /run_pytorch_training.py --batch_size 768 --checkpoint_all_hosts False --disablecache True --epochs 100 --freeze_conv_layers True --image_process_pipeline CaffeRCNNTransformationPipelineEdgePad --learning_rate 0.0002 --loss_criteria CustomCrossEntropyLoss --model_architecture MaskRAlexnet --output_size 4"
2022-06-02 17:23:01,168 [INFO ] Received keyword args: {'learning_rate': 0.0002, 'momentum': None, 'weight_decay': None, 'batch_size': 768, 'train_ratio': None, 'step_size': None, 'gamma': None, 'finetune': None, 'disablecache': True, 'profile_training': None, 'epochs': 100, 'output_size': 4, 'model_architecture': 'MaskRAlexnet', 'optimizer_type': None, 'loss_criteria': 'CustomCrossEntropyLoss', 'freeze_conv_layers': True, 'image_process_pipeline': 'CaffeRCNNTransformationPipelineEdgePad', 'epoch_length_factor': 1.0, 'checkpoint_all_hosts': False, 'parameter_config_file': None}
2022-06-02 17:23:01,

Start with code/EamesJointEmbedding/src/eames_joint_embedding/train/trigger_train.py, especially launch_training and the estimator.fit call, then inspect the SageMaker training-job logs for the underlying ExecuteUserScriptError. Done means identifying the cause of the failed training job and confirming that the reported training command completes without UnexpectedStatusException.

Written by the indexing model from the issue text.