aws / aws/sagemaker-python-sdk

Can you support a parameter to increase shared memory bus?

Open
#5,592 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

Hi,

I am on version 3.4.1 on python 3.12.
I am running into errors like
```
RuntimeError: DataLoader worker (pid 256) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.
```

As far as I understand, this means the workers are using too much memory transferring it to the main process, and this error is being raised. The usual fix would be just to increase the memory bus limit, but it seems that sagemaker does not support such a parameter: https://stackoverflow.com/questions/71660619/how-to-alter-shared-memory-for-sagemaker-docker-containers (at least, that's the only result I found, from 4 years ago)

Would it be possible to add such a capability? It would help a lot - right now I am manually reducing the batch size just to fix this problem.

Thank you!

Contributor guide

Open the contributing guide

Research direction

No repository file, test, or entry point is named. Start by locating SageMaker training-container configuration and existing runtime parameters, then determine where a shared-memory setting could be exposed; done means the parameter is supported and its behavior is covered by tests or documented usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, pytorch
Domain
cloud, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.