aws-samples / aws-samples/sagemaker-studio-custom-image-samples

Custom images launched in SageMaker Studio use Python installation not present in uploaded image

Open
#9 6 comments 7 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
136
Forks
90
PR merge metrics
No merged PRs in 30d

Description

## Expected Behavior

1. I attach a custom docker container to my SageMaker Studio session ([example Dockerfile](https://github.com/singlecellopenproblems/SingleCellOpenProblems/blob/docker-jupyter-kernels/docker/openproblems/Dockerfile))
2. I launch a SageMaker Studio notebook using a Jupyter Kernel from the image
3. The kernel launches the system Python installation at `/usr/local/bin` referencing packages installed at `/usr/local/lib/python3.8/site-packages`

## Οbserved behavior
The SageMaker Studio notebook launches a kernel session using a python installation **not present in the Docker container when uploaded to ECR** found at `/opt/.sagemakerinternal/conda/bin/python`.

This happens with Notebook and Console kernel sessions launched from the image.

## Attempted debugging
I can confirm that the Notebook and Console sessions are launching the correct image because I can find my installed packages in `/usr/local/lib/python3.8/site-packages` from within the Notebook and the Console.

I can confirm this doesn't happen if I launch an **Image Terminal** from the Studio Session. When I do this, the Docker container is launched as expected and executing `$ python` points to the expected installation in `/usr/local/bin`

I think this has something to do with how Jupyter sessions are launched within SageMaker Studio. When I look at the logs on CloudWatch, I see this:

![image (4)](https://user-images.githubusercontent.com/8322751/111911071-51338800-8a3a-11eb-83a0-4be068819ce6.png)

I have a few concerns:
1. Why is SageMaker using conda here when my image doesn't have conda installed? Can I override this? I would have thought the main purpose of Custom images was to be able to use a different version of Python than standard SageMaker Images. Why override this when importing custom images?
2. What does it mean that this warning is getting raised: `+ echo This is not a DLC/Studio image (found Custom), so not adding Python3 kernel.` This should be a SageMaker image. I make it copying the `echo_kernel` example Dockerfile from this repo. SageMaker has no issues launching the Notebook session from the console.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.