dbader / dbader/schedule

How to execute jobs in parallel potential Stackoverflow ?

Open
#292 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
12.3k
Forks
999
PR merge metrics
No merged PRs in 30d

Description

Hi @dbader ,

First of all thank you for creating this also package, it is very useful and versatile.

I have been using the following feature for little over a year now,until recently I have noticed some issue:
https://schedule.readthedocs.io/en/stable/faq.html#how-to-execute-jobs-in-parallel

Recently, the number of thread that I'm running has been increased(it was about 10-20 pending thread, now it about 40+ pending thread), then the `schedule.run_pending()` will not kick off any task afterwards, more strangely, there is no error or exception has been through, it felt that the whole method call been silenced/skipped.

I did some initial investigation, noticed the thread limit might be the issue, not the number of thread but the `stack size`. I'm running it on MacOS, the default stack size is about 8Mb, the max size the OS allowed is 65532K, which is what I set for now, after change the Stack Size to the Max. the issue seems went away at the moment.

So, my question goes, is there a potential stack overflow issue while doing multi-threading with many task running in parallel ? If is, why it didn't blow up... with limited knowledge of system programming, I'm hoping someone know about it can help answer it.

System: Mac OS 10.14.3
Python: 3.7.2
Schedule Version: 0.6.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the documented parallel-execution approach in the FAQ and the schedule.run_pending() entry point. Reproduce the behavior on macOS 10.14.3 with Python 3.7.2, Schedule 0.6.0, and 40+ pending threads while comparing stack-size settings; done means determining whether the library is responsible and documenting or resolving the observed silent stop.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.