Gabriella439 / Gabriella439/Haskell-Pipes-Safe-Library

onException seems to use plenty of memory

Open
#46 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
27
Forks
22
Avg merge
12d 10h
Merged PRs (30d)
1

Description

Hello, I am writing a program which read its data from an hdf5 file. for each images of this file it compute soimething and add it into an IORef wchi is is in fact a C objects managed via FFI

the interesting part of the code is here

```
processHkl :: FramesHklP a => InputHkl a b -> IO ()
processHkl input@(InputHkl det _ h5d o res cen d r config' mask') = do
pixels <- getPixelsCoordinates det cen d r

jobs <- mkJobsHkl input
r'<- mapConcurrently (\job -> withCubeAccumulator $ \s -> do
runSafeT $ runEffect $
each job
>-> tee Pipes.Prelude.print
>-> framesHklP h5d det
>-> Pipes.Prelude.mapM (liftIO . spaceHkl config' det pixels res mask')
>-> mkCube'P det s
) jobs
Prelude.print r'
return ()
-- saveCube o r'

```

Pipe Safe is used to managed the resources, (open and close files), In my case I open 10 times the same file (for test)
My program use a lot's of memory and it seems that most of this memory is in the onException method.

here part of the profilling informations. all the hkl_xxx are FFI functions (time consuming)

```
Mon Oct 19 17:23 2020 Time and Allocation Profiling Report (Final)

binoculars-ng +RTS -N -s -pa -RTS process data/test/config_sixs_local.ini

total time = 289.35 secs (730656 ticks @ 1000 us, 24 processors)
total alloc = 3,469,792,304 bytes (excludes profiling overheads)

COST CENTRE MODULE SRC %time %alloc ticks bytes

IDLE IDLE 41.4 0.0 302154 0
hkl_binoculars_space_hkl Hkl.Binoculars.Projections src/Hkl/Binoculars/Projections.hs:188:53-146 23.5 0.1 171493 2085248
hkl_binoculars_cube_new_merge' Hkl.C.Binoculars src/Hkl/C/Binoculars.hsc:85:61-96 13.8 0.0 100607 0
SYSTEM SYSTEM 10.4 0.0 76271 1472776
hkl_binoculars_cube_new' Hkl.Binoculars.Common src/Hkl/Binoculars/Common.hs:136:53-119 4.8 0.0 34870 794408
GC GC 3.7 0.0 27336 1112
onException Pipes.Safe src/Pipes/Safe.hs:(400,1)-(404,12) 0.9 52.9 6565 1834549720
h5e_try Bindings.HDF5.Raw.H5E src/Bindings/HDF5/Raw/H5E.hsc:(387,1)-(391,17) 0.7 9.3 5042 322654064
control Control.Monad.Trans.Control Control/Monad/Trans/Control.hs:751:1-39 0.4 10.2 2569 352506304
for Pipes src/Pipes.hs:176:1-11 0.1 3.4 911 118529496
```

So I would like your help in order to investigate this issue.

This program should use a constant amount of memory independently of the number of images red. But this is not the case, the more image, the bigger the onException is.

thanks for considering

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the profiling report and the onException definition in src/Pipes/Safe.hs:400-404, then reproduce the issue using the process data command described in the report. Compare memory usage as the number of images increases and inspect the related h5e_try and Control.Monad.Trans.Control entries. Done means identifying why onException retains increasing allocations and confirming constant memory usage across image counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.