hydrusnetwork / hydrusnetwork/hydrus
URL converter not honoured for File urls
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 207
- PR merge metrics
- No merged PRs in 30d
Description
Nitter has been coughing up blood lately, throwing 504s (because its app backend is responding to nginx too slowly). Nitter seems to want to serve all media by downloading and then forwarding it. To see if I could mitigate the bandwidth problem I was trying to see if I could alter the nitter parsers to use twitter as CDN for media, so that nitter only needs to worry about serving a structure hydrus knows how to parse, not any of the corresponding media files.
Simple enough I thought, I'll just change the nitter file url class to convert to the twitter CDN, make sure there is a twitter file url class installed and it should just work right?
Not so it seems. Here is my attempt

Nitter media
[50, "nitter tweet media", 9, ["4201c88c64d355d91282f83a59a6e85babf6214b4a616f69e2816b0442c1d231", 2, "https", "nitter.net", [false, false, false, true, true, false], [[[51, 1, [0, "pic", null, null, "pic"]], null], [[51, 1, [2, "media%2F.*", null, null, "media%2FEZv1zeRVcAEtBnz.jpg%3Fname%3Dorig"]], null]], [], [55, 1, [[[5, "url percent encoding"], [9, ["https://nitter.net/pic", "https://pbs.twimg.com"]]], "https://nitter.net/pic/media%2FEZv1zeRVcAEtBnz.jpg%3Fname%3Dorig"]], 0, [55, 1, [[], "https://hostname.com/post/page.php?id=123456&s=view"]], null, null, 1, "https://nitter.net/pic/media%2FEZv1zeRVcAEtBnz.jpg%3Fname%3Dorig"]]
Twitter media
[50, "twitter media", 9, ["efc50e7beb9aa006b00a35c2dc0a5c9eb3660c6b2d392f2382d761e39e6d700e", 2, "https", "pbs.twimg.com", [false, false, true, false, true, false], [[[51, 1, [0, "media", null, null, "media"]], null], [[51, 1, [3, "", null, null, "example string"]], null]], [], [55, 1, [[], "https://hostname.com/post/page.php?id=123456&s=view"]], 0, [55, 1, [[], "https://hostname.com/post/page.php?id=123456&s=view"]], null, null, 1, "https://pbs.twimg.com/media/ElEN80eUcAAShil?format=jpg&name=orig"]]
But when I put a twitter tweet url into a test url download page, it still tried to serve the media through nitter.
I have a strong suspicion that nitter is dying because some or several hydrus users are asking for way too much, so I really hopped that would work and we could just roll that change into the defaults a future version.
Environment
Hydrus version:414
Platform: Mac 10.14
OS specifics:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how hydrus handles URL converters for File URLs, using the Nitter and Twitter media examples in the issue. Compare the converter behavior with the test URL download page and determine why the Nitter URL is still served instead of being converted to the Twitter CDN. Done means the configured conversion is honored for this case and the resulting media URL is used.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100