aws / aws/amazon-sagemaker-examples

[Bug Report]

Open
#3,736 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
[Add the link to the notebook.](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-pipelines/tabular/local-mode/sagemaker-pipelines-local-mode.ipynb)

**Describe the bug**
This notebooks fails at the command `execution = pipeline.start()`with the error:
Aborting on container exit...
Pipeline step 'AbaloneTrain' FAILED. Failure message is: RuntimeError: Failed to run: ['docker-compose', '-f', '/tmp/tmpiqqpn04g/docker-compose.yaml', 'up', '--build', '--abort-on-container-exit'], Process exited with code: 1
Pipeline execution 41bd9d52-fffd-416f-afe2-fc7edaf2f355 FAILED because step 'AbaloneTrain' failed.

**To reproduce**
You can see this error if you just run the whole notebook by clicking run all. I got this error for all p37 and p38 kernels

**Logs**
If applicable, add logs to help explain your problem.
You may also attach an `.ipynb` file to this issue if it includes relevant logs or output.
Here is the full logs
Creating x6v8sbtoit-algo-1-mshpt ...
Creating x6v8sbtoit-algo-1-mshpt ... done
Attaching to x6v8sbtoit-algo-1-mshpt
x6v8sbtoit-algo-1-mshpt | [2023-01-12 15:16:04.427 41160b0e496b:1 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None
x6v8sbtoit-algo-1-mshpt | [2023-01-12:15:16:04:INFO] Imported framework sagemaker_xgboost_container.training
x6v8sbtoit-algo-1-mshpt | [2023-01-12:15:16:04:INFO] Failed to parse hyperparameter objective value reg:squarederror to Json.
x6v8sbtoit-algo-1-mshpt | Returning the value itself
x6v8sbtoit-algo-1-mshpt | [2023-01-12:15:16:04:INFO] No GPUs detected (normal if no gpus installed)
x6v8sbtoit-algo-1-mshpt | [2023-01-12:15:16:04:INFO] Invoking user training script.
x6v8sbtoit-algo-1-mshpt | [2023-01-12:15:16:04:ERROR] Reporting training FAILURE
x6v8sbtoit-algo-1-mshpt | [2023-01-12:15:16:04:ERROR] framework error:
x6v8sbtoit-algo-1-mshpt | Traceback (most recent call last):
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/sagemaker_containers/_trainer.py", line 84, in train
x6v8sbtoit-algo-1-mshpt | entrypoint()
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/sagemaker_xgboost_container/training.py", line 102, in main
x6v8sbtoit-algo-1-mshpt | train(framework.training_env())
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/sagemaker_xgboost_container/training.py", line 87, in train
x6v8sbtoit-algo-1-mshpt | framework.modules.run_module(
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/sagemaker_containers/_modules.py", line 290, in run_module
x6v8sbtoit-algo-1-mshpt | _files.download_and_extract(uri, _env.code_dir)
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/sagemaker_containers/_files.py", line 129, in download_and_extract
x6v8sbtoit-algo-1-mshpt | s3_download(uri, dst)
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/sagemaker_containers/_files.py", line 165, in s3_download
x6v8sbtoit-algo-1-mshpt | s3.Bucket(bucket).download_file(key, dst)
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/boto3/s3/inject.py", line 244, in bucket_download_file
x6v8sbtoit-algo-1-mshpt | return self.meta.client.download_file(
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/boto3/s3/inject.py", line 170, in download_file
x6v8sbtoit-algo-1-mshpt | return transfer.download_file(
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/boto3/s3/transfer.py", line 307, in download_file
x6v8sbtoit-algo-1-mshpt | future.result()
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
x6v8sbtoit-algo-1-mshpt | return self._coordinator.result()
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
x6v8sbtoit-algo-1-mshpt | raise self._exception
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/s3transfer/tasks.py", line 255, in _main
x6v8sbtoit-algo-1-mshpt | self._submit(transfer_future=transfer_future, **kwargs)
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/s3transfer/download.py", line 340, in _submit
x6v8sbtoit-algo-1-mshpt | response = client.head_object(
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call
x6v8sbtoit-algo-1-mshpt | return self._make_api_call(operation_name, kwargs)
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/botocore/client.py", line 676, in _make_api_call
x6v8sbtoit-algo-1-mshpt | raise error_class(parsed_response, operation_name)
x6v8sbtoit-algo-1-mshpt | botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
x6v8sbtoit-algo-1-mshpt |
x6v8sbtoit-algo-1-mshpt | An error occurred (403) when calling the HeadObject operation: Forbidden
x6v8sbtoit-algo-1-mshpt exited with code 1
1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.