internetarchive / internetarchive/warcprox
--profile doesn't work with Python 3.12+
- Dominant language
- Python
- Stars
- 467
- Forks
- 66
- PR merge metrics
- No merged PRs in 30d
Description
Python 3.12 introduced a new limitation preventing multiple nested profilers from running at the same time: https://github.com/python/cpython/issues/110770
We're running into this limitation with warcprox. Attempting to run with `--profile` enabled will cause it to immediately error out with an `Another profiling tool is already active` exception the first time it tries to spin up a `BasePostfetchProcessor` thread.
We instantiate `cProfiler` profilers in one place - inside `BasePostfetchProcessor.run`: https://github.com/internetarchive/warcprox/blob/37e4cd50930ed35d929114a5558fb40430de942e/warcprox/__init__.py#L106-L109
We do, however, additionally *interact* with those profilers via `PooledMitmProxy`, where we grab the profiler from the thread, enable it, process the request, and disable it again: https://github.com/internetarchive/warcprox/blob/f7dd1e75af021bdc106d94491c12c6825beb6a61/warcprox/mitmproxy.py#L753-L756
This could either be repeated invocations to `enable()` the same profiler when it's already enabled (calling `.enable()` twice on the same profiler raises an exception), or a call to `enable()` a second profiler from a different thread while a profiler from the first thread is still active.
Contributor guide
No contributing guide indexed for this repository
Research direction
Inspect BasePostfetchProcessor.run in warcprox/__init__.py and the profiler interactions in PooledMitmProxy in warcprox/mitmproxy.py. Reproduce the --profile failure on Python 3.12+, then trace when each thread enables and disables its profiler. Done means --profile runs without the nested-profiler exception while processing requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100