hydrusnetwork / hydrusnetwork/hydrus

Hydrus treats throttling error doc as temporary repeated parse error, instead of serious bandwidth error when parsing JSON

Open
#784 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug system:downloading system:network
Dominant language
Python
Stars
3.2k
Forks
207
No merged PRs in 30d

Description

When hydrus expects a JSON parse but receives an error document in HTML because of throttelling, it gets into an exception loop of expecting JSON, instead of properly backing off.

Environment

Hydrus version:414 (Source)
Platform: Mac 10.13
OS specifics:

What happens

When hydrus expects a JSON parse but receives an error document in HTML because of throttelling, it gets into an exception loop of expecting JSON, instead of properly backing off.

Steps to reproduce

1.Run an instagram sub aggressively

Desired outcome

Hydrus backs off exponentially allowing instagram to calm down and stop detecting it as malicious traffic, instead of Hydrus looking like a DDoS or just skiddy bot.

Error message / Log file / Screenshots

Traceback Log 

Traceback (most recent call last):
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 2419, in Parse
    whole_page_parse_results.extend( content_parser.Parse( parsing_context, converted_parsing_text ) )
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 2074, in Parse
    raise e
hydrus.core.HydrusExceptions.ParseException: Content Parser image url: Unable to parse that JSON: Expecting value: line 1 column 1 (char 0). JSON sample: <!DOCTYPE html>
<html lang="en" class="no-js not-logged-in client-root">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">

        <title>
Login • Instagram
</title>

        
        <meta name="robots" content="noimageindex, noarchive">
        <meta name="apple-mobile-web-app-status-bar-style" content="default">
        <meta name="mobile-web-app-capable" content="yes">
        <meta name="theme-color" content="#ffffff">
        <meta id="viewport" name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, viewport-fit=cover">
        <link rel="manifest" href="/data/manifest.json">

        <link rel="preload" href="/static/bundles/es6/ConsumerUICommons.css/01aad45fe5a8.css" as="style" type="text/css" crossorigin="anonymous" />
<link rel="preload" href="/static/bundles/es6/ConsumerAsyncCommons.css/0608bd6190e0.css" as="style" type="text/css" crossorigin="anonymous" />
<link rel="preload" href="/static/bundles/es6/C

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/importing/ClientImportFileSeeds.py", line 1185, in WorkOnURL
    all_parse_results = parser.Parse( parsing_context, parsing_text )
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 2433, in Parse
    raise e
hydrus.core.HydrusExceptions.ParseException: Page Parser instagram post api parser: Content Parser image url: Unable to parse that JSON: Expecting value: line 1 column 1 (char 0). JSON sample: <!DOCTYPE html>
<html lang="en" class="no-js not-logged-in client-root">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">

        <title>
Login • Instagram
</title>

        
        <meta name="robots" content="noimageindex, noarchive">
        <meta name="apple-mobile-web-app-status-bar-style" content="default">
        <meta name="mobile-web-app-capable" content="yes">
        <meta name="theme-color" content="#ffffff">
        <meta id="viewport" name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, viewport-fit=cover">
        <link rel="manifest" href="/data/manifest.json">

        <link rel="preload" href="/static/bundles/es6/ConsumerUICommons.css/01aad45fe5a8.css" as="style" type="text/css" crossorigin="anonymous" />
<link rel="preload" href="/static/bundles/es6/ConsumerAsyncCommons.css/0608bd6190e0.css" as="style" type="text/css" crossorigin="anonymous" />
<link rel="preload" href="/static/bundles/es6/C

2021/01/13 07:03:23: Error when processing https://www.instagram.com/p/CH_XmaunkoT!
2021/01/13 07:03:23: Traceback (most recent call last):
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 1712, in _ParseRawTexts
    j = HG.client_controller.parsing_cache.GetJSON( parsing_text )
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientCaches.py", line 386, in GetJSON
    json_object = json.loads( json_text )
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 2066, in Parse
    parsed_texts = list( self._formula.Parse( parsing_context, parsing_text ) )
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 615, in Parse
    raw_texts = self._ParseRawTexts( parsing_context, parsing_text )
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 1718, in _ParseRawTexts
    raise HydrusExceptions.ParseException( message )
hydrus.core.HydrusExceptions.ParseException: Unable to parse that JSON: Expecting value: line 1 column 1 (char 0). JSON sample: <!DOCTYPE html>
<html lang="en" class="no-js not-logged-in client-root">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">

        <title>
Login • Instagram
</title>

        
        <meta name="robots" content="noimageindex, noarchive">
        <meta name="apple-mobile-web-app-status-bar-style" content="default">
        <meta name="mobile-web-app-capable" content="yes">
        <meta name="theme-color" content="#ffffff">
        <meta id="viewport" name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, viewport-fit=cover">
        <link rel="manifest" href="/data/manifest.json">

        <link rel="preload" href="/static/bundles/es6/ConsumerUICommons.css/01aad45fe5a8.css" as="style" type="text/css" crossorigin="anonymous" />
<link rel="preload" href="/static/bundles/es6/ConsumerAsyncCommons.css/0608bd6190e0.css" as="style" type="text/css" crossorigin="anonymous" />
<link rel="preload" href="/static/bundles/es6/C

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 2419, in Parse
    whole_page_parse_results.extend( content_parser.Parse( parsing_context, converted_parsing_text ) )
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 2074, in Parse
    raise e
hydrus.core.HydrusExceptions.ParseException: Content Parser image url: Unable to parse that JSON: Expecting value: line 1 column 1 (char 0). JSON sample: <!DOCTYPE html>
<html lang="en" class="no-js not-logged-in client-root">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">

        <title>
Login • Instagram
</title>

        
        <meta name="robots" content="noimageindex, noarchive">
        <meta name="apple-mobile-web-app-status-bar-style" content="default">
        <meta name="mobile-web-app-capable" content="yes">
        <meta name="theme-color" content="#ffffff">
        <meta id="viewport" name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, viewport-fit=cover">
        <link rel="manifest" href="/data/manifest.json">

        <link rel="preload" href="/static/bundles/es6/ConsumerUICommons.css/01aad45fe5a8.css" as="style" type="text/css" crossorigin="anonymous" />
<link rel="preload" href="/static/bundles/es6/ConsumerAsyncCommons.css/0608bd6190e0.css" as="style" type="text/css" crossorigin="anonymous" />
<link rel="preload" href="/static/bundles/es6/C

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/importing/ClientImportFileSeeds.py", line 1185, in WorkOnURL
    all_parse_results = parser.Parse( parsing_context, parsing_text )
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 2433, in Parse
    raise e
hydrus.core.HydrusExceptions.ParseException: Page Parser instagram post api parser: Content Parser image url: Unable to parse that JSON: Expecting value: line 1 column 1 (char 0). JSON sample: <!DOCTYPE html>
<html lang="en" class="no-js not-logged-in client-root">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">

        <title>
Login • Instagram
</title>

        
        <meta name="robots" content="noimageindex, noarchive">
        <meta name="apple-mobile-web-app-status-bar-style" content="default">
        <meta name="mobile-web-app-capable" content="yes">
        <meta name="theme-color" content="#ffffff">
        <meta id="viewport" name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, viewport-fit=cover">
        <link rel="manifest" href="/data/manifest.json">

        <link rel="preload" href="/static/bundles/es6/ConsumerUICommons.css/01aad45fe5a8.css" as="style" type="text/css" crossorigin="anonymous" />
<link rel="preload" href="/static/bundles/es6/ConsumerAsyncCommons.css/0608bd6190e0.css" as="style" type="text/css" crossorigin="anonymous" />
<link rel="preload" href="/static/bundles/es6/C

2021/01/13 07:05:52: Error when processing https://www.instagram.com/p/CIHN2B5HwPf!
2021/01/13 07:05:52: Traceback (most recent call last):
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 1712, in _ParseRawTexts
    j = HG.client_controller.parsing_cache.GetJSON( parsing_text )
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientCaches.py", line 386, in GetJSON
    json_object = json.loads( json_text )
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 2066, in Parse
    parsed_texts = list( self._formula.Parse( parsing_context, parsing_text ) )
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 615, in Parse
    raw_texts = self._ParseRawTexts( parsing_context, parsing_text )
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 1718, in _ParseRawTexts
    raise HydrusExceptions.ParseException( message )
hydrus.core.HydrusExceptions.ParseException: Unable to parse that JSON: Expecting value: line 1 column 1 (char 0). JSON sample: <!DOCTYPE html>
<html lang="en" class="no-js not-logged-in client-root">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">

        <title>
Login • Instagram
</title>

        
        <meta name="robots" content="noimageindex, noarchive">
        <meta name="apple-mobile-web-app-status-bar-style" content="default">
        <meta name="mobile-web-app-capable" content="yes">
        <meta name="theme-color" content="#ffffff">
        <meta id="viewport" name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, viewport-fit=cover">
        <link rel="manifest" href="/data/manifest.json">

        <link rel="preload" href="/static/bundles/es6/ConsumerUICommons.css/01aad45fe5a8.css" as="style" type="text/css" crossorigin="anonymous" />
<link rel="preload" href="/static/bundles/es6/ConsumerAsyncCommons.css/0608bd6190e0.css" as="style" type="text/css" crossorigin="anonymous" />
<link rel="preload" href="/static/bundles/es6/C

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 2419, in Parse
    whole_page_parse_results.extend( content_parser.Parse( parsing_context, converted_parsing_text ) )
  File "/Users/redacted/git/hydrus/hydrusqt-userpatch/hydrus/client/ClientParsing.py", line 2074, in Parse
    raise e
hydrus.core.HydrusExceptions.ParseException: Content Parser image url: Unable to parse that JSON: Expecting value: line 1 column 1 (char 0). JSON sample: <!DOCTYPE html>
<html lang="en" class="no-js not-logged-in client-root">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">

        <title>
Login • Instagram
</title>
#...

</p>
</details>

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the reported failure through hydrus/client/ClientParsing.py, especially Parse and _ParseRawTexts, then inspect hydrus/client/ClientCaches.py:GetJSON and hydrus/client/importing/ClientImportFileSeeds.py:WorkOnURL. Reproduce the aggressive Instagram subscription case and verify that an HTML throttling response is treated as a bandwidth error, backs off exponentially, and does not repeat the JSON parse loop.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.