aws / aws/amazon-sagemaker-examples

Sagemaker processing job with BYOC, unable to start sklearn_preprocessor.transform from within container.

Open
#3,430 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

I'm currently deploying a Jupyter Notebook inside a docker container via this github: https://github.com/aws-samples/sagemaker-run-notebook

Sagemaker creates a 'processing job' which launches my docker container and runs the jupyter notebook within it via papermill.

However sagemaker Processing Jobs don't have access it seems to any internal AWS services despite operating within a VPC. This is normal behaviour and you need to create VPC Endpoints within your VPC that provides access for services within the VPC to specific AWS internal API's. Here's an example: "com.amazonaws.eu-west-2.s3" which provides access for a processing job to S3.

So, my problem is when I run my script it hangs/timesout on this cell:

```
transformer = sklearn_preprocessor.transformer(
instance_count=1,
instance_type='ml.m4.xlarge',
assemble_with='Line',
accept='text/csv',
output_path = f"s3://{BUCKET}/{PROJECT_NAME}/preprocessor/")
transformer.transform(train_inputs[0], content_type='text/csv')

```
I'm assuming that this .transform function is calling some internal service that I haven't built and endpoint for.

And ideas?

No log output from Papermill as the cell execution just hangs.

Contributor guide

Open the contributing guide

Research direction

Start with the notebook cell that calls sklearn_preprocessor.transformer and then transformer.transform, and review the SageMaker Processing Job and VPC endpoint configuration described in the issue. Check the Papermill execution behavior and available logs around the timeout. Done means the transform call completes from the container without hanging and the notebook records successful processing output.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python, scikit-learn
Domain
cloud, machine-learning, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.