internetarchive / internetarchive/warcprox

warcproxy context manager?

Open
#18 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
467
Forks
66
PR merge metrics
No merged PRs in 30d

Description

Hi,

I've used warcproxy indirectly through the perma project who (as you probably know) do the phantomjs + warcproxy dance to create archives.

While reading and modifying the code I noticed that the proxy usage pattern almost perfectly matches the use case of context managers:
- set up background scaffolding (the proxy)
- hand over a handle to the relevant context variables (a class instance or at least the CA file location and the ip:port address)
- pull down everything once finished (join the threads)

Would you consider adding such a context manager to the warcproxy project? Adding it here should be the best fit, in case the class API would need to be modified.

PS: cc @jcushman since their code might benefit from this (hope you don't mind the ping)

A rough sketch of the idea looks like this (pasted together from perma.cc code and simplified, not actually runnable):

```
@contextmanager
def warc_proxy(*args, **kwargs):
"""
Context manager for warcproxy
"""
# Set up proxy instance
# use kwargs with default arguments
proxy = WarcProxy(server_address=('127.0.0.1'),
kwargs.get('port', 27500),
recorded_url_q=some_q,
)
writer_thread = WarcWriterThread(recorded_url_q=some_q)
proxy.warcprox_controller = WarcproxController(proxy, writer_thread)
proxy.warcprox_thread = threading.Thread(target=proxy.warcprox_controller.run_until_shutdown)
proxy.warcprox_thread.start()

try:
# whatever we are yielding would need to carry all relevant data
# such as adding the threads as instance attributes
yield proxy
finally:
# tear down
proxy.warcprox_controller.stop.set()
proxy.warcprox_thread.join()
```

edit: Ah, and here is a simple usage example:

```
with warc_proxy(port=5000) as proxy:
browser = setup_browser(ca=proxy.ca.ca_file, address=proxy.server_address)
browser.do_stuff()
# proxy with all threads disappears at scope exit
```

Now if that's not tidy I don't know what is!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating WarcProxy, WarcWriterThread, and WarcproxController, then trace the existing proxy startup and shutdown flow. The requested context manager should expose the proxy data needed by the browser and cleanly stop and join its threads when the with block exits.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, networking
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.