Worker concurrency
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 95
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 1
Description
Currently, a TaskTiger worker can process only one task concurrently. Is it possible to have one worker prefork multiple pool subprocesses so that it could process multiple tasks at same time?
In our case, we need preload some big modules(long time to load and a significant amount of memory usage) in parent process for performance. Starting multiple workers consumes too much memory, and this problem could be fixed if TaskTiger support worker concurrency.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the TaskTiger worker entry point and its current single-task process model. Define how one worker should prefork multiple pool subprocesses while preloading large modules in the parent, then verify that multiple tasks run concurrently without requiring separate workers and that the stated memory concern is addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100