hydrusnetwork / hydrusnetwork/hydrus

URL converter not honoured for File urls

Open
#697 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
3.2k
Forks
207
PR merge metrics
No merged PRs in 30d

Description

Nitter has been coughing up blood lately, throwing 504s (because its app backend is responding to nginx too slowly). Nitter seems to want to serve all media by downloading and then forwarding it. To see if I could mitigate the bandwidth problem I was trying to see if I could alter the nitter parsers to use twitter as CDN for media, so that nitter only needs to worry about serving a structure hydrus knows how to parse, not any of the corresponding media files.

Simple enough I thought, I'll just change the nitter file url class to convert to the twitter CDN, make sure there is a twitter file url class installed and it should just work right?

Not so it seems. Here is my attempt
nitter file converrt

Nitter media
[50, "nitter tweet media", 9, ["4201c88c64d355d91282f83a59a6e85babf6214b4a616f69e2816b0442c1d231", 2, "https", "nitter.net", [false, false, false, true, true, false], [[[51, 1, [0, "pic", null, null, "pic"]], null], [[51, 1, [2, "media%2F.*", null, null, "media%2FEZv1zeRVcAEtBnz.jpg%3Fname%3Dorig"]], null]], [], [55, 1, [[[5, "url percent encoding"], [9, ["https://nitter.net/pic", "https://pbs.twimg.com"]]], "https://nitter.net/pic/media%2FEZv1zeRVcAEtBnz.jpg%3Fname%3Dorig"]], 0, [55, 1, [[], "https://hostname.com/post/page.php?id=123456&s=view"]], null, null, 1, "https://nitter.net/pic/media%2FEZv1zeRVcAEtBnz.jpg%3Fname%3Dorig"]]
Twitter media
[50, "twitter media", 9, ["efc50e7beb9aa006b00a35c2dc0a5c9eb3660c6b2d392f2382d761e39e6d700e", 2, "https", "pbs.twimg.com", [false, false, true, false, true, false], [[[51, 1, [0, "media", null, null, "media"]], null], [[51, 1, [3, "", null, null, "example string"]], null]], [], [55, 1, [[], "https://hostname.com/post/page.php?id=123456&s=view"]], 0, [55, 1, [[], "https://hostname.com/post/page.php?id=123456&s=view"]], null, null, 1, "https://pbs.twimg.com/media/ElEN80eUcAAShil?format=jpg&name=orig"]]

But when I put a twitter tweet url into a test url download page, it still tried to serve the media through nitter.

I have a strong suspicion that nitter is dying because some or several hydrus users are asking for way too much, so I really hopped that would work and we could just roll that change into the defaults a future version.

Environment

Hydrus version:414
Platform: Mac 10.14
OS specifics:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how hydrus handles URL converters for File URLs, using the Nitter and Twitter media examples in the issue. Compare the converter behavior with the test URL download page and determine why the Nitter URL is still served instead of being converted to the Twitter CDN. Done means the configured conversion is honored for this case and the resulting media URL is used.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.