aws / aws/sagemaker-training-toolkit

Bash Command ENTRYPOINT Expects `train` argument

Open
#65 8 comments 1 reaction 0 assignees View on GitHub
type: question
Dominant language
Python
Stars
530
Forks
140
Avg merge
1h 12m
Merged PRs (30d)
2

Description

**Describe the bug**
I would like to create a SageMaker Training Job using a custom Docker container which executes a bash command I have created. I am using the kedro framework to organise and structure my code into pipelines and nodes. I would like to execute my training code with the bash command
```bash
kedro run --tag train_pipeline
```

For some reason, Sagemaker passes `train` as a default execution parameter.

**To reproduce**
The following is my Dockerfile:

```
FROM python:3.7-stretch

# install project requirements
COPY src/requirements_sm.txt /tmp/requirements.txt
RUN pip install -r /tmp/requirements.txt && rm -f /tmp/requirements.txt

# install nltk dependencies
RUN python -m nltk.downloader punkt

# Copy the whole project except what is in .dockerignore
COPY . /opt/ml/code

# Set working directory
WORKDIR /opt/ml/code

ENTRYPOINT ["kedro", "run"]

```

I am creating and running a sagemaker job with the following code:

```python
hyperparams = {
'tag': 'train_pipeline',
}

estimator = Estimator(
image_name=IMAGE_NAME,
role=IAM_ROLE,
train_instance_count=1,
train_instance_type='local',
tags=TAGS,
subnets=SUBNETS,
security_group_ids=SG_IDS,
hyperparameters=hyperparams,
output_kms_key=KMS_KEY,
output_path=BUCKET_PATH
)

estimator.fit()
```

When execute the estimator.fit() I get the following error:
```
Creating tmpvixmgk5s_algo-1-p8xwk_1 ... done
Attaching to tmpvixmgk5s_algo-1-p8xwk_1
algo-1-p8xwk_1 | Usage: kedro run [OPTIONS]
algo-1-p8xwk_1 | Try 'kedro run -h' for help.
algo-1-p8xwk_1 |
algo-1-p8xwk_1 | Error: Got unexpected extra argument (train)
tmpvixmgk5s_algo-1-p8xwk_1 exited with code 2
```

Why does Sagemaker pass a train argument by default to the bash command?

**Expected behavior**
I would like expect the sagemaker job to execute the following bash command within the job:
```
kedro run --tag train_pipeline
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing how estimator.fit() launches the Dockerfile ENTRYPOINT and appends the train argument. Reproduce the behavior with the shown Dockerfile and hyperparameters, then verify that the documented or supported invocation allows the requested kedro run --tag train_pipeline command to execute without the unexpected argument.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, python
Domain
cloud, devops, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.