aws / aws/amazon-sagemaker-examples

Error for training job failed. reason: algorithmerror: exit code: 127

Open
#1,589 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

Hello,

Same to #969

I was training a DistilBERT model on SageMaker instance using fast-bert. I am using the ml.p2.xlarge instance for GPU processing.

When the function downloads the training image from ECR during **fit()**, I happen to receive _**"/usr/bin/env: ‘python\r’: No such file or directory".**_ See below -

![image](https://user-images.githubusercontent.com/42617598/94992908-e33df600-05aa-11eb-80c0-83833c0c139d.png)

And, at the end of stack-trace received the following - _**error for training job failed. reason: algorithmerror: exit code: 127**_

![image](https://user-images.githubusercontent.com/42617598/94992943-17191b80-05ab-11eb-9233-7909ca666140.png)

**Tech Stack-**

[fast-bert](https://github.com/kaushaltrivedi/fast-bert/tree/master/fast_bert) docker image
SageMaker NB Instance - ml.t2.medium
GPU Compute - ml.p2.xlarge

What could be the reason for this error? My IAM role has all the required permissions.

Kindly help.

Contributor guide

Open the contributing guide

Research direction

Start by comparing this report with #969 and tracing the fast-bert Docker image invocation during SageMaker fit(). Confirm whether the reported `/usr/bin/env` failure is reproducible on the stated instances, then document the verified cause and resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.