Explore subinterpreter based gunicorn worker for achieving parallelism
- Dominant language
- Python
- Stars
- 66
- Forks
- 23
- PR merge metrics
- No merged PRs in 30d
Description
We currently use `gthread` worker to serve multiple requests with single process. This however, is severely limited by GIL so we need to add multiple processes to achieve true parallelism. Processes are expensive as they can only share very small portion of code in memory (pre-fork)
Python now supports running multiple sub-interpreters each with their own "GIL". Figure out a way to write a gunicorn worker that can use subinterpreters instead of threads to achieve both parallelism and concurrency without paying significantly extra memory costs.
Note: To implement a POC for this you'll have to use python 3.14 as that's the only version that has Python bindings for sub-interpreters, older versions only exposed C bindings AFAIK.
Relevant reads that you might find interesting:
- https://docs.gunicorn.org/en/latest/design.html
- https://peps.python.org/pep-0734/
- https://docs.python.org/3.14/library/concurrent.futures.html#concurrent.futures.InterpreterPoolExecutor
- https://github.com/frappe/gunicorn/blob/bb554053bb87218120d76ab6676af7015680e8b6/gunicorn/workers/gthread.py#L66
- https://realpython.com/python-gil/
- (old but conceptually relevant) https://www.youtube.com/watch?v=Obt-vMVdM8s
- (Frappe context) https://frappe.io/blog/engineering/reducing-memory-footprint-of-frappe-framework
- (frappe context) https://github.com/frappe/press/pull/1218
- (conceptually relevant) https://www.youtube.com/watch?v=WqrCnVAkLIo
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with gunicorn/workers/gthread.py and the Gunicorn design documentation, then review Python 3.14's InterpreterPoolExecutor and PEP 734. A completed proof of concept should provide a subinterpreter-based worker and demonstrate parallel request handling without the memory cost of multiple processes; compare its behavior with the existing gthread worker.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100