aws / aws/sagemaker-training-toolkit
sagemaker container doesn't support running container image with more privilege
- Dominant language
- Python
- Stars
- 530
- Forks
- 140
- Avg merge
- 1h 12m
- Merged PRs (30d)
- 2
Description
It is assumed that Sagemaker runs docker image (custom image) with no privilege. This the reason why some of the custom images are unable to run with Sagemaker as they required more permission to interact with the host hardware such as /dev/.
One example is that when having a custom image to enable s3fs inside the container, Sagemaker was unable to run s3fs features as /dev/fuse was not permitted from the host.
When running the image with ``docker run`` outside SM with ``--cap-add SYS_ADMIN --device /dev/fuse`` or with ``--privileged``, the container is able to access /dev/fuse.
However, when running the same image on Sagemaker as a part of training job, the container was unable to provide access to the device (/dev/fuse) hence s3fs functionality.
See details about docker run enabling extras functionality https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
How to fix: Adding a param to make sure when loading an SM image (custom), developers are able to select add-on privilege if required. See https://docs.docker.com/compose/compose-file/
Contributor guide
Assessment
This issue has not been assessed yet.