Gabriella439 / Gabriella439/Haskell-Succinct-Vector-Library

Indexing a 78MB bit-string blows up memory

Open
#2 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
20
Forks
0
PR merge metrics
No merged PRs in 30d

Description

I encountered the following problem while using the library.

The following sequence of commands in `ghci` will start thrashing.

```
:set -XBangPatterns
import Data.Vector
import System.IO.MMap
import Data.Word
import Data.Vector.Storable.ByteString
import qualified Data.Vector.Storable as VS
fp <- mmapFileByteString "78mb.json" Nothing
let !v = (byteStringToVector fp) :: VS.Vector Word64
let !w = convert v :: Vector Word64
let !bv = prepare w
```

I replaced code for `v1` with dummy vector:

```
v1 :: Unboxed.Vector Int
v1 = Unboxed.singleton 1
-- flip Unboxed.snoc (len * 64)
-- ( Unboxed.map (\(_, i) -> i)
-- ( Unboxed.filter (\(j, _) -> j `rem` 512 == 0)
-- ( Unboxed.imap (,)
-- ( oneIndices 0 (Unboxed.length v)
-- v ))))
```

And the problem went away.

Note that calling `oneIndices 0 (Unboxed.length w) w` alone is sufficient to blow up memory.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.