aws / aws/amazon-sagemaker-examples
[Bug Report]
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
[Add the link to the notebook.](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-pipelines/tabular/local-mode/sagemaker-pipelines-local-mode.ipynb)
**Describe the bug**
This notebooks fails at the command `execution = pipeline.start()`with the error:
Aborting on container exit...
Pipeline step 'AbaloneTrain' FAILED. Failure message is: RuntimeError: Failed to run: ['docker-compose', '-f', '/tmp/tmpiqqpn04g/docker-compose.yaml', 'up', '--build', '--abort-on-container-exit'], Process exited with code: 1
Pipeline execution 41bd9d52-fffd-416f-afe2-fc7edaf2f355 FAILED because step 'AbaloneTrain' failed.
**To reproduce**
You can see this error if you just run the whole notebook by clicking run all. I got this error for all p37 and p38 kernels
**Logs**
If applicable, add logs to help explain your problem.
You may also attach an `.ipynb` file to this issue if it includes relevant logs or output.
Here is the full logs
Creating x6v8sbtoit-algo-1-mshpt ...
Creating x6v8sbtoit-algo-1-mshpt ... done
Attaching to x6v8sbtoit-algo-1-mshpt
x6v8sbtoit-algo-1-mshpt | [2023-01-12 15:16:04.427 41160b0e496b:1 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None
x6v8sbtoit-algo-1-mshpt | [2023-01-12:15:16:04:INFO] Imported framework sagemaker_xgboost_container.training
x6v8sbtoit-algo-1-mshpt | [2023-01-12:15:16:04:INFO] Failed to parse hyperparameter objective value reg:squarederror to Json.
x6v8sbtoit-algo-1-mshpt | Returning the value itself
x6v8sbtoit-algo-1-mshpt | [2023-01-12:15:16:04:INFO] No GPUs detected (normal if no gpus installed)
x6v8sbtoit-algo-1-mshpt | [2023-01-12:15:16:04:INFO] Invoking user training script.
x6v8sbtoit-algo-1-mshpt | [2023-01-12:15:16:04:ERROR] Reporting training FAILURE
x6v8sbtoit-algo-1-mshpt | [2023-01-12:15:16:04:ERROR] framework error:
x6v8sbtoit-algo-1-mshpt | Traceback (most recent call last):
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/sagemaker_containers/_trainer.py", line 84, in train
x6v8sbtoit-algo-1-mshpt | entrypoint()
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/sagemaker_xgboost_container/training.py", line 102, in main
x6v8sbtoit-algo-1-mshpt | train(framework.training_env())
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/sagemaker_xgboost_container/training.py", line 87, in train
x6v8sbtoit-algo-1-mshpt | framework.modules.run_module(
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/sagemaker_containers/_modules.py", line 290, in run_module
x6v8sbtoit-algo-1-mshpt | _files.download_and_extract(uri, _env.code_dir)
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/sagemaker_containers/_files.py", line 129, in download_and_extract
x6v8sbtoit-algo-1-mshpt | s3_download(uri, dst)
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/sagemaker_containers/_files.py", line 165, in s3_download
x6v8sbtoit-algo-1-mshpt | s3.Bucket(bucket).download_file(key, dst)
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/boto3/s3/inject.py", line 244, in bucket_download_file
x6v8sbtoit-algo-1-mshpt | return self.meta.client.download_file(
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/boto3/s3/inject.py", line 170, in download_file
x6v8sbtoit-algo-1-mshpt | return transfer.download_file(
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/boto3/s3/transfer.py", line 307, in download_file
x6v8sbtoit-algo-1-mshpt | future.result()
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
x6v8sbtoit-algo-1-mshpt | return self._coordinator.result()
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
x6v8sbtoit-algo-1-mshpt | raise self._exception
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/s3transfer/tasks.py", line 255, in _main
x6v8sbtoit-algo-1-mshpt | self._submit(transfer_future=transfer_future, **kwargs)
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/s3transfer/download.py", line 340, in _submit
x6v8sbtoit-algo-1-mshpt | response = client.head_object(
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call
x6v8sbtoit-algo-1-mshpt | return self._make_api_call(operation_name, kwargs)
x6v8sbtoit-algo-1-mshpt | File "/miniconda3/lib/python3.8/site-packages/botocore/client.py", line 676, in _make_api_call
x6v8sbtoit-algo-1-mshpt | raise error_class(parsed_response, operation_name)
x6v8sbtoit-algo-1-mshpt | botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
x6v8sbtoit-algo-1-mshpt |
x6v8sbtoit-algo-1-mshpt | An error occurred (403) when calling the HeadObject operation: Forbidden
x6v8sbtoit-algo-1-mshpt exited with code 1
1
Contributor guide
Assessment
This issue has not been assessed yet.