aws / aws/sagemaker-training-toolkit
Bash Command ENTRYPOINT Expects `train` argument
- Dominant language
- Python
- Stars
- 530
- Forks
- 140
- Avg merge
- 1h 12m
- Merged PRs (30d)
- 2
Description
**Describe the bug**
I would like to create a SageMaker Training Job using a custom Docker container which executes a bash command I have created. I am using the kedro framework to organise and structure my code into pipelines and nodes. I would like to execute my training code with the bash command
```bash
kedro run --tag train_pipeline
```
For some reason, Sagemaker passes `train` as a default execution parameter.
**To reproduce**
The following is my Dockerfile:
```
FROM python:3.7-stretch
# install project requirements
COPY src/requirements_sm.txt /tmp/requirements.txt
RUN pip install -r /tmp/requirements.txt && rm -f /tmp/requirements.txt
# install nltk dependencies
RUN python -m nltk.downloader punkt
# Copy the whole project except what is in .dockerignore
COPY . /opt/ml/code
# Set working directory
WORKDIR /opt/ml/code
ENTRYPOINT ["kedro", "run"]
```
I am creating and running a sagemaker job with the following code:
```python
hyperparams = {
'tag': 'train_pipeline',
}
estimator = Estimator(
image_name=IMAGE_NAME,
role=IAM_ROLE,
train_instance_count=1,
train_instance_type='local',
tags=TAGS,
subnets=SUBNETS,
security_group_ids=SG_IDS,
hyperparameters=hyperparams,
output_kms_key=KMS_KEY,
output_path=BUCKET_PATH
)
estimator.fit()
```
When execute the estimator.fit() I get the following error:
```
Creating tmpvixmgk5s_algo-1-p8xwk_1 ... done
Attaching to tmpvixmgk5s_algo-1-p8xwk_1
algo-1-p8xwk_1 | Usage: kedro run [OPTIONS]
algo-1-p8xwk_1 | Try 'kedro run -h' for help.
algo-1-p8xwk_1 |
algo-1-p8xwk_1 | Error: Got unexpected extra argument (train)
tmpvixmgk5s_algo-1-p8xwk_1 exited with code 2
```
Why does Sagemaker pass a train argument by default to the bash command?
**Expected behavior**
I would like expect the sagemaker job to execute the following bash command within the job:
```
kedro run --tag train_pipeline
```
Contributor guide
Research direction
Start by tracing how estimator.fit() launches the Dockerfile ENTRYPOINT and appends the train argument. Reproduce the behavior with the shown Dockerfile and hyperparameters, then verify that the documented or supported invocation allows the requested kedro run --tag train_pipeline command to execute without the unexpected argument.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, python
- Domain
- cloud, devops, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100