get for UArray blows the heap for large arrays
- 主要语言
- Haskell
- 星标
- 120
- 派生
- 70
- PR 合并指标
- 30 天内没有已合并 PR
描述
``` haskell
instance (Binary i, Ix i, Binary e, IArray UArray e) => Binary (UArray i e) where
get = do
bs <- get
n <- get
xs <- getMany n
return (listArray bs xs)
```
getMany is fully strict in the list, since it uses an accumulator and reverses it at the end. The intermediate xs list can be huge in cases where the eventual UArray is much more manageable (eg 28M Booleans).
Two questions:
1) Is there a known alternative for (un)serializing UArrays to(from) disk? Such an alternative would make this Issue far less important.
2) Have you considered a version that serializes the bytes directly? I drafted one up; it's tremendously more efficient, though I'm concerned about robustness wrt endianness etc. Furthermore, it requires a base monad that can mutate arrays, which requires an "unsafe" invocation. And lastly it's not portable, using ghc-prim.
HTH. Thanks.
贡献指南
这个仓库没有索引到贡献指南
调研方向
从 Binary (UArray i e) 的 get 实例及其 getMany 路径开始,复现大型 Boolean UArray 的解码,以测量中间列表的内存使用情况。根据字节序、可变性、ghc-prim 的可移植性和磁盘序列化替代方案,审查提议的 direct-byte 方法。完成的标准是确定一致同意的范围,并拥有一条经过测试、可避免堆内存暴增的 UArray 序列化路径。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- haskell
- 领域
- data
- Issue 类型
- 缺陷
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 25/100