codelucas / codelucas/newspaper

Using build() in monkeypatched script sometimes produces infinite loop

Open
#751 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.2k
Forks
2.1k
PR merge metrics
No merged PRs in 30d

Description

I'm attempting to use Newspaper within a Flask app, which needs to be monkeypatched to perform some other functions. However, when the library is monkeypatched, rss feeds cannot be read without raising a RecursionError.

Minimal example:

from eventlet import monkey_patch
from newspaper import Article, build, popular_urls
import logging

logging.basicConfig(filename=__name__, level=20)

def test():
	for x in popular_urls()[:5]:
		try:
			paper = build(x, memoize_articles=False)
			logging.info(f'Found {len(paper.articles)} articles for {x}')
		except RecursionError as RE:
			logging.warning(RE)

if __name__ == '__main__':
	test()
	monkey_patch()
	logging.info('Monkey Patched!')
	test()

Logging output:

INFO:root:Found 0 articles for http://www.huffingtonpost.com
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: Not Found for url: http://edition.cnn.com/feeds
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: Not Found for url: http://edition.cnn.com/feed
INFO:root:Found 875 articles for http://cnn.com
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: Not Found for url: https://time.com/feeds/
INFO:root:Found 0 articles for http://www.time.com
CRITICAL:newspaper.network:[REQUEST FAILED] Failed to parse: https://support.ted.com 
WARNING:newspaper.source:Deleting category https://support.ted.com  from source http://www.ted.com due to download error
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: Not Found for url: https://www.ted.com/feeds
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: Not Found for url: https://www.ted.com/feed
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: Not Found for url: https://www.ted.com/rss
INFO:root:Found 152 articles for http://www.ted.com
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: OK for url: https://pando.com/feeds/
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: OK for url: https://pando.com/rss/
INFO:root:Found 2 articles for http://pandodaily.com
INFO:root:Monkey Patched!
WARNING:root:maximum recursion depth exceeded
WARNING:root:maximum recursion depth exceeded
WARNING:root:maximum recursion depth exceeded
WARNING:root:maximum recursion depth exceeded
WARNING:root:maximum recursion depth exceeded

Traceback:

Traceback (most recent call last):
  File "C:\Users\james.ITR\OneDrive - International Tin Association\Python\ita\ita\portal\flask_test.py", line 16, in <module>
    test()
  File "C:\Users\james.ITR\OneDrive - International Tin Association\Python\ita\ita\portal\flask_test.py", line 9, in test
    paper = build(x, memoize_articles=False)
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\newspaper\api.py", line 30, in build
    s.build()
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\newspaper\source.py", line 91, in build
    self.download()
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\newspaper\source.py", line 180, in download
    self.html = network.get_html(self.url, self.config)
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\newspaper\network.py", line 41, in get_html
    return get_html_2XX_only(url, config, response)
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\newspaper\network.py", line 63, in get_html_2XX_only
    url=url, **get_request_kwargs(timeout, useragent, proxies, headers))
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 668, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 668, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 247, in resolve_redirects
    **adapter_kwargs
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 344, in _make_request
    self._validate_conn(conn)
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 843, in _validate_conn
    conn.connect()
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connection.py", line 350, in connect
    cert_reqs=resolve_cert_reqs(self.cert_reqs),
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\ssl_.py", line 280, in create_urllib3_context
    context.options |= options
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 507, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 507, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 507, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  [Previous line repeated 485 more times]
RecursionError: maximum recursion depth exceeded

Seems to be an issue in requests' with certificates and options. Is there any way to fix this or avoid monkeypatching this specific library?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with newspaper/network.py and source.py, then trace the minimal example through requests, urllib3, and Python's ssl.py as shown in the traceback. Reproduce the first run and the run after eventlet.monkey_patch(); done means the patched run no longer reaches recursive SSLContext.options calls or raises RecursionError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.