internetarchive / internetarchive/warcprox

--profile doesn't work with Python 3.12+

Open
#258 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
467
Forks
66
PR merge metrics
No merged PRs in 30d

Description

Python 3.12 introduced a new limitation preventing multiple nested profilers from running at the same time: https://github.com/python/cpython/issues/110770

We're running into this limitation with warcprox. Attempting to run with `--profile` enabled will cause it to immediately error out with an `Another profiling tool is already active` exception the first time it tries to spin up a `BasePostfetchProcessor` thread.

We instantiate `cProfiler` profilers in one place - inside `BasePostfetchProcessor.run`: https://github.com/internetarchive/warcprox/blob/37e4cd50930ed35d929114a5558fb40430de942e/warcprox/__init__.py#L106-L109

We do, however, additionally *interact* with those profilers via `PooledMitmProxy`, where we grab the profiler from the thread, enable it, process the request, and disable it again: https://github.com/internetarchive/warcprox/blob/f7dd1e75af021bdc106d94491c12c6825beb6a61/warcprox/mitmproxy.py#L753-L756

This could either be repeated invocations to `enable()` the same profiler when it's already enabled (calling `.enable()` twice on the same profiler raises an exception), or a call to `enable()` a second profiler from a different thread while a profiler from the first thread is still active.

Contributor guide

No contributing guide indexed for this repository

Research direction

Inspect BasePostfetchProcessor.run in warcprox/__init__.py and the profiler interactions in PooledMitmProxy in warcprox/mitmproxy.py. Reproduce the --profile failure on Python 3.12+, then trace when each thread enables and disables its profiler. Done means --profile runs without the nested-profiler exception while processing requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.