haskell / haskell/binary

get for UArray blows the heap for large arrays

Open
#34 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
120
Forks
70
PR merge metrics
No merged PRs in 30d

Description

``` haskell
instance (Binary i, Ix i, Binary e, IArray UArray e) => Binary (UArray i e) where
get = do
bs <- get
n <- get
xs <- getMany n
return (listArray bs xs)
```

getMany is fully strict in the list, since it uses an accumulator and reverses it at the end. The intermediate xs list can be huge in cases where the eventual UArray is much more manageable (eg 28M Booleans).

Two questions:

1) Is there a known alternative for (un)serializing UArrays to(from) disk? Such an alternative would make this Issue far less important.

2) Have you considered a version that serializes the bytes directly? I drafted one up; it's tremendously more efficient, though I'm concerned about robustness wrt endianness etc. Furthermore, it requires a base monad that can mutate arrays, which requires an "unsafe" invocation. And lastly it's not portable, using ghc-prim.

HTH. Thanks.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Binary (UArray i e) get instance and its getMany path, reproducing decoding of a large Boolean UArray to measure the intermediate list's memory use. Review the proposed direct-byte approach against endianness, mutability, ghc-prim portability, and disk serialization alternatives. Done means an agreed scope and a tested UArray serialization path that avoids the heap blow-up.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
data
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.