aws-samples / aws-samples/sagemaker-studio-custom-image-samples
Custom images launched in SageMaker Studio use Python installation not present in uploaded image
- Dominant language
- No language data
- Stars
- 136
- Forks
- 90
- PR merge metrics
- No merged PRs in 30d
Description
## Expected Behavior
1. I attach a custom docker container to my SageMaker Studio session ([example Dockerfile](https://github.com/singlecellopenproblems/SingleCellOpenProblems/blob/docker-jupyter-kernels/docker/openproblems/Dockerfile))
2. I launch a SageMaker Studio notebook using a Jupyter Kernel from the image
3. The kernel launches the system Python installation at `/usr/local/bin` referencing packages installed at `/usr/local/lib/python3.8/site-packages`
## Οbserved behavior
The SageMaker Studio notebook launches a kernel session using a python installation **not present in the Docker container when uploaded to ECR** found at `/opt/.sagemakerinternal/conda/bin/python`.
This happens with Notebook and Console kernel sessions launched from the image.
## Attempted debugging
I can confirm that the Notebook and Console sessions are launching the correct image because I can find my installed packages in `/usr/local/lib/python3.8/site-packages` from within the Notebook and the Console.
I can confirm this doesn't happen if I launch an **Image Terminal** from the Studio Session. When I do this, the Docker container is launched as expected and executing `$ python` points to the expected installation in `/usr/local/bin`
I think this has something to do with how Jupyter sessions are launched within SageMaker Studio. When I look at the logs on CloudWatch, I see this:

I have a few concerns:
1. Why is SageMaker using conda here when my image doesn't have conda installed? Can I override this? I would have thought the main purpose of Custom images was to be able to use a different version of Python than standard SageMaker Images. Why override this when importing custom images?
2. What does it mean that this warning is getting raised: `+ echo This is not a DLC/Studio image (found Custom), so not adding Python3 kernel.` This should be a SageMaker image. I make it copying the `echo_kernel` example Dockerfile from this repo. SageMaker has no issues launching the Notebook session from the console.
Contributor guide
Assessment
This issue has not been assessed yet.