Gabriella439 / Gabriella439/turtle
Task-pooling based parallelization & incremental output
- Dominant language
- Haskell
- Stars
- 978
- Forks
- 94
- PR merge metrics
- No merged PRs in 30d
Description
When scheduling longer tasks (like audio file conversion), all cores should be used by the script.
Right now Turtle only contains `parallel`, which will schedule as many parallel processes as `fork` allows and only return the results after every element has been processed.
I’d like to use something like [`async-pool`](https://hackage.haskell.org/package/async-pool-0.9.0.1/docs/Control-Concurrent-Async-Pool.html#v:mapTasks), here’s a first try:
```haskell
finished <- liftIO $ Pool.withTaskGroup 4 $ \tg ->
Pool.mapTasks tg $ map action infiles
select finished >>= printf s
```
It uses exactly four cores now, but the output is still deferred until the very end; I’m not sure if it’s possible to integrate that into the streaming abstraction?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading Turtle's existing `parallel` functionality and its streaming abstraction, then compare them with the linked `async-pool` `mapTasks` example. The work is complete when task concurrency is bounded while results can be emitted incrementally rather than deferred until all inputs finish.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100