verify_url fails on relative redirects and causes RecursionError on loops
- Dominant language
- Python
- Stars
- 451
- Forks
- 707
- Avg merge
- 22h 59m
- Merged PRs (30d)
- 91
Description
**Description**
The `verify_url` method in `backend/apps/owasp/scraper.py` has two flaws in its redirect handling logic:
* **Infinite Recursion:** There is no depth limit or cycle detection. If a server redirects `A -> B -> A`, the method calls itself indefinitely until it raises a `RecursionError`.
* **Relative Redirect Failure:** The method does not resolve relative `Location` headers (e.g., `/login`). It passes the relative path directly to `urlparse(url).netloc`, which returns an empty string, causing the scraper to incorrectly mark valid URLs as invalid (`return None`).
**Expected Behavior**
* The scraper should stop following redirects after a reasonable limit (e.g., 30 hops) to prevent crashes.
* It should correctly resolve relative redirects (e.g., converting `/about` to `http://test.com/about`) using `urllib.parse.urljoin`.
**Proposed Fix**
I propose updating `verify_url` to:
* Accept a `depth` parameter to track recursion depth.
* Use `urljoin(url, location)` to normalize redirect URLs before the recursive call.
I have a fix ready and can submit a PR.
**Are you going to work on fixing this?**
- [X] Yes
- [ ] No
Contributor guide
Assessment
This issue has not been assessed yet.