aws / aws/sagemaker-training-toolkit

sagemaker container doesn't support running container image with more privilege

Open
#21 1 comment 0 reactions 0 assignees View on GitHub
type: enhancement
Dominant language
Python
Stars
530
Forks
140
Avg merge
1h 12m
Merged PRs (30d)
2

Description

It is assumed that Sagemaker runs docker image (custom image) with no privilege. This the reason why some of the custom images are unable to run with Sagemaker as they required more permission to interact with the host hardware such as /dev/.

One example is that when having a custom image to enable s3fs inside the container, Sagemaker was unable to run s3fs features as /dev/fuse was not permitted from the host.

When running the image with ``docker run`` outside SM with ``--cap-add SYS_ADMIN --device /dev/fuse`` or with ``--privileged``, the container is able to access /dev/fuse.

However, when running the same image on Sagemaker as a part of training job, the container was unable to provide access to the device (/dev/fuse) hence s3fs functionality.

See details about docker run enabling extras functionality https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities

How to fix: Adding a param to make sure when loading an SM image (custom), developers are able to select add-on privilege if required. See https://docs.docker.com/compose/compose-file/

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.