aws / aws/amazon-sagemaker-examples
Sagemaker processing job with BYOC, unable to start sklearn_preprocessor.transform from within container.
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
I'm currently deploying a Jupyter Notebook inside a docker container via this github: https://github.com/aws-samples/sagemaker-run-notebook
Sagemaker creates a 'processing job' which launches my docker container and runs the jupyter notebook within it via papermill.
However sagemaker Processing Jobs don't have access it seems to any internal AWS services despite operating within a VPC. This is normal behaviour and you need to create VPC Endpoints within your VPC that provides access for services within the VPC to specific AWS internal API's. Here's an example: "com.amazonaws.eu-west-2.s3" which provides access for a processing job to S3.
So, my problem is when I run my script it hangs/timesout on this cell:
```
transformer = sklearn_preprocessor.transformer(
instance_count=1,
instance_type='ml.m4.xlarge',
assemble_with='Line',
accept='text/csv',
output_path = f"s3://{BUCKET}/{PROJECT_NAME}/preprocessor/")
transformer.transform(train_inputs[0], content_type='text/csv')
```
I'm assuming that this .transform function is calling some internal service that I haven't built and endpoint for.
And ideas?
No log output from Papermill as the cell execution just hangs.
Contributor guide
Research direction
Start with the notebook cell that calls sklearn_preprocessor.transformer and then transformer.transform, and review the SageMaker Processing Job and VPC endpoint configuration described in the issue. Check the Papermill execution behavior and available logs around the timeout. Done means the transform call completes from the container without hanging and the notebook records successful processing output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, python, scikit-learn
- Domain
- cloud, machine-learning, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100