Gabriella439 / Gabriella439/turtle

Task-pooling based parallelization & incremental output

Open
#252 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
978
Forks
94
PR merge metrics
No merged PRs in 30d

Description

When scheduling longer tasks (like audio file conversion), all cores should be used by the script.
Right now Turtle only contains `parallel`, which will schedule as many parallel processes as `fork` allows and only return the results after every element has been processed.

I’d like to use something like [`async-pool`](https://hackage.haskell.org/package/async-pool-0.9.0.1/docs/Control-Concurrent-Async-Pool.html#v:mapTasks), here’s a first try:

```haskell
finished <- liftIO $ Pool.withTaskGroup 4 $ \tg ->
Pool.mapTasks tg $ map action infiles
select finished >>= printf s
```

It uses exactly four cores now, but the output is still deferred until the very end; I’m not sure if it’s possible to integrate that into the streaming abstraction?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading Turtle's existing `parallel` functionality and its streaming abstraction, then compare them with the linked `async-pool` `mapTasks` example. The work is complete when task concurrency is bounded while results can be emitted incrementally rather than deferred until all inputs finish.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.