aws / aws/sagemaker-python-sdk

Can you support a parameter to increase shared memory bus?

未关闭
#5,592 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
2.3k
派生
1.3k
平均合并
1 天 22 小时
30 天内合并 PR
35

描述

Hi,

I am on version 3.4.1 on python 3.12.
I am running into errors like
```
RuntimeError: DataLoader worker (pid 256) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.
```

As far as I understand, this means the workers are using too much memory transferring it to the main process, and this error is being raised. The usual fix would be just to increase the memory bus limit, but it seems that sagemaker does not support such a parameter: https://stackoverflow.com/questions/71660619/how-to-alter-shared-memory-for-sagemaker-docker-containers (at least, that's the only result I found, from 4 years ago)

Would it be possible to add such a capability? It would help a lot - right now I am manually reducing the batch size just to fix this problem.

Thank you!

贡献指南

打开贡献指南

调研方向

未指定任何仓库文件、测试或入口点。首先定位 SageMaker 训练容器配置和现有运行时参数,然后确定可以在哪里暴露共享内存设置;当该参数得到支持,且其行为有测试或文档化用法覆盖时,即视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
aws, python, pytorch
领域
cloud, machine-learning
Issue 类型
功能
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。