codelucas / codelucas/newspaper
Using build() in monkeypatched script sometimes produces infinite loop
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.2k
- Forks
- 2.1k
- PR merge metrics
- No merged PRs in 30d
Description
I'm attempting to use Newspaper within a Flask app, which needs to be monkeypatched to perform some other functions. However, when the library is monkeypatched, rss feeds cannot be read without raising a RecursionError.
Minimal example:
from eventlet import monkey_patch
from newspaper import Article, build, popular_urls
import logging
logging.basicConfig(filename=__name__, level=20)
def test():
for x in popular_urls()[:5]:
try:
paper = build(x, memoize_articles=False)
logging.info(f'Found {len(paper.articles)} articles for {x}')
except RecursionError as RE:
logging.warning(RE)
if __name__ == '__main__':
test()
monkey_patch()
logging.info('Monkey Patched!')
test()
Logging output:
INFO:root:Found 0 articles for http://www.huffingtonpost.com
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: Not Found for url: http://edition.cnn.com/feeds
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: Not Found for url: http://edition.cnn.com/feed
INFO:root:Found 875 articles for http://cnn.com
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: Not Found for url: https://time.com/feeds/
INFO:root:Found 0 articles for http://www.time.com
CRITICAL:newspaper.network:[REQUEST FAILED] Failed to parse: https://support.ted.com
WARNING:newspaper.source:Deleting category https://support.ted.com from source http://www.ted.com due to download error
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: Not Found for url: https://www.ted.com/feeds
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: Not Found for url: https://www.ted.com/feed
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: Not Found for url: https://www.ted.com/rss
INFO:root:Found 152 articles for http://www.ted.com
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: OK for url: https://pando.com/feeds/
CRITICAL:newspaper.network:[REQUEST FAILED] 404 Client Error: OK for url: https://pando.com/rss/
INFO:root:Found 2 articles for http://pandodaily.com
INFO:root:Monkey Patched!
WARNING:root:maximum recursion depth exceeded
WARNING:root:maximum recursion depth exceeded
WARNING:root:maximum recursion depth exceeded
WARNING:root:maximum recursion depth exceeded
WARNING:root:maximum recursion depth exceeded
Traceback:
Traceback (most recent call last):
File "C:\Users\james.ITR\OneDrive - International Tin Association\Python\ita\ita\portal\flask_test.py", line 16, in <module>
test()
File "C:\Users\james.ITR\OneDrive - International Tin Association\Python\ita\ita\portal\flask_test.py", line 9, in test
paper = build(x, memoize_articles=False)
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\newspaper\api.py", line 30, in build
s.build()
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\newspaper\source.py", line 91, in build
self.download()
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\newspaper\source.py", line 180, in download
self.html = network.get_html(self.url, self.config)
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\newspaper\network.py", line 41, in get_html
return get_html_2XX_only(url, config, response)
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\newspaper\network.py", line 63, in get_html_2XX_only
url=url, **get_request_kwargs(timeout, useragent, proxies, headers))
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 668, in send
history = [resp for resp in gen] if allow_redirects else []
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 668, in <listcomp>
history = [resp for resp in gen] if allow_redirects else []
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 247, in resolve_redirects
**adapter_kwargs
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 603, in urlopen
chunked=chunked)
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 344, in _make_request
self._validate_conn(conn)
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 843, in _validate_conn
conn.connect()
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connection.py", line 350, in connect
cert_reqs=resolve_cert_reqs(self.cert_reqs),
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\ssl_.py", line 280, in create_urllib3_context
context.options |= options
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 507, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 507, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "C:\Users\james.ITR\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 507, in options
super(SSLContext, SSLContext).options.__set__(self, value)
[Previous line repeated 485 more times]
RecursionError: maximum recursion depth exceeded
Seems to be an issue in requests' with certificates and options. Is there any way to fix this or avoid monkeypatching this specific library?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with newspaper/network.py and source.py, then trace the minimal example through requests, urllib3, and Python's ssl.py as shown in the traceback. Reproduce the first run and the run after eventlet.monkey_patch(); done means the patched run no longer reaches recursive SSLContext.options calls or raises RecursionError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100