Unintuitive out of memory errors, and odd OOMs
未关闭
还没有人认领这个 Issue。
- 主要语言
- Java
- 星标
- 928
- 派生
- 227
- PR 合并指标
- 30 天内没有已合并 PR
描述
I'm experimenting with TensorScope, and thus using large amounts of memory for tensors. However, when I try this, I get odd oom-related errors.
Example (kotlin):
repeat(1000) {
TFloat32.tensorOf(Shape.of(10000, 10000))
}
gives
2021-01-11 20:36:45.599992: W external/org_tensorflow/tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 400000000 exceeds 10% of free system memory.
2021-01-11 20:36:45.625897: W external/org_tensorflow/tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 400000000 exceeds 10% of free system memory.
2021-01-11 20:36:45.626092: W external/org_tensorflow/tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 400000000 exceeds 10% of free system memory.
2021-01-11 20:36:45.626180: W external/org_tensorflow/tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 400000000 exceeds 10% of free system memory.
2021-01-11 20:36:45.626262: W external/org_tensorflow/tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 400000000 exceeds 10% of free system memory.
Exception in thread "main" java.lang.NullPointerException
at org.tensorflow.internal.buffer.TensorBuffers.tensorMemory(TensorBuffers.java:180)
at org.tensorflow.internal.buffer.TensorBuffers.toFloats(TensorBuffers.java:102)
at org.tensorflow.internal.types.TFloat32Mapper.mapDense(TFloat32Mapper.java:34)
at org.tensorflow.internal.types.TFloat32Mapper.mapDense(TFloat32Mapper.java:30)
at org.tensorflow.RawTensor.asTypedTensor(RawTensor.java:215)
at org.tensorflow.Tensor.of(Tensor.java:90)
at org.tensorflow.Tensor.of(Tensor.java:63)
at org.tensorflow.types.TFloat32.tensorOf(TFloat32.java:79)
at tester.TensorScopeKt.main(TensorScope.kt:8)
at tester.TensorScopeKt.main(TensorScope.kt)
I'd think this is a fairly straight forward OOM, just without nice error messages, but the used memory in Task Manager (I'm on Windows) barely moves. I do start using heap memory, but it caps 100MB, even with -Xmx6g. Is there some limit on tensor allocations I'm not aware of?
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
首先通过 TFloat32.tensorOf 重现 Kotlin 示例,并跟踪 org/tensorflow/internal/buffer/TensorBuffers.java:180 处报告的失败。将 native 分配警告与 Java 端异常进行比较,并将 done 定义为:使用准确且可操作的错误报告分配失败,而不是出现所观察到的 NullPointerException。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java, kotlin
- 领域
- machine-learning
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 35/100