windmill-labs / windmill-labs/windmill
feature: support for async def main in python
Open
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 18k
- Forks
- 1.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 328
Description
Not entirely sure how this should work or if it's a use case you want to support.
Problem:
- I have a long-running script that takes 10 files as input; most of the time is spent waiting for network requests
- the first step of that script is to run OCR on each file: send the file to a OCR service and wait until it's processed, then return the text; this takes e.g. 10 minutes
- the OCR is in a separate script
- I'm currently running everything in a main python script that imports the OCR script and asyncio.gather s the 10 files
- this works fine, but:
- the first 10 minutes of runtime are spent waiting for the api calls to return, and the worker is blocked during that time even though it's just waiting on API calls
- I can't run the ocr script as a separate job, with
wmill.run_script, which would e.g. let me take advantage of built-in cache features, without tying up an extra worker per file for 10 minutes
Proposal:
- windmill scripts with async def main could run concurrently on one worker. The worker would run the eventloop.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing how Python scripts with async def main are currently executed and how wmill.run_script assigns work to workers. Compare that behavior with the proposed worker event loop and concurrent execution model. Done should mean async scripts can share one worker while waiting on network requests, without requiring a separate worker per file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100