haskell / haskell/binary

get for UArray blows the heap for large arrays

未关闭
#34 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Haskell
星标
120
派生
70
PR 合并指标
30 天内没有已合并 PR

描述

``` haskell
instance (Binary i, Ix i, Binary e, IArray UArray e) => Binary (UArray i e) where
get = do
bs <- get
n <- get
xs <- getMany n
return (listArray bs xs)
```

getMany is fully strict in the list, since it uses an accumulator and reverses it at the end. The intermediate xs list can be huge in cases where the eventual UArray is much more manageable (eg 28M Booleans).

Two questions:

1) Is there a known alternative for (un)serializing UArrays to(from) disk? Such an alternative would make this Issue far less important.

2) Have you considered a version that serializes the bytes directly? I drafted one up; it's tremendously more efficient, though I'm concerned about robustness wrt endianness etc. Furthermore, it requires a base monad that can mutate arrays, which requires an "unsafe" invocation. And lastly it's not portable, using ghc-prim.

HTH. Thanks.

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 Binary (UArray i e) 的 get 实例及其 getMany 路径开始,复现大型 Boolean UArray 的解码,以测量中间列表的内存使用情况。根据字节序、可变性、ghc-prim 的可移植性和磁盘序列化替代方案,审查提议的 direct-byte 方法。完成的标准是确定一致同意的范围,并拥有一条经过测试、可避免堆内存暴增的 UArray 序列化路径。

由索引模型根据 Issue 内容生成。

评估

技术栈
haskell
领域
data
Issue 类型
缺陷
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。