hypothesis / hypothesis/via

Return 404 instead of 400 responses for obviously-invalid URLs

Open
#1,434 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29
Forks
13
PR merge metrics
No merged PRs in 30d

Description

Requests for "obviously invalid" URLs like `https://via.hypothes.is/wp-admin` return 400 responses instead of 404. This is inconvenient because we cannot easily filter out such responses in eg. New Relic metrics which monitor the overall error rate of the service.

We have encountered situations when a bot hits a large number of URLs like this in a short window of time, typically looking for vulnerabilities in common PHP packages. This triggered an alarm that fires when 80%+ of the service's requests are failing for a period of time (10-15 minutes).

The reason for the 400 here is that `/wp-admin` matches the general route for proxying websites which treats the part after the initial `/` as a URL, where the protocol is optional. `CheckmateClient.check_url` fails to parse `wp-admin` as a public URL and raises `BadURL`, which results in a 400 response.

For context, see https://hypothes-is.slack.com/archives/C074BUPEG/p1728300410941439?thread_ts=1728292002.576029&cid=C074BUPEG.

New Relic alert: https://one.newrelic.com/alerts/issue?account=1385283&duration=259200000&state=e0b2c426-026d-27ee-4aa8-b0894fb965d1

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the general proxy route that handles paths such as `/wp-admin` and follow its call to `CheckmateClient.check_url`. Reproduce the request and inspect how `BadURL` becomes a 400 response. Done means obviously invalid proxy URLs return 404 without changing valid URL handling; add or update a regression test if the existing test location is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.