OWASP / OWASP/Nest

verify_url fails on relative redirects and causes RecursionError on loops

Open
#3,969 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
451
Forks
707
Avg merge
22h 59m
Merged PRs (30d)
91

Description

**Description**
The `verify_url` method in `backend/apps/owasp/scraper.py` has two flaws in its redirect handling logic:

* **Infinite Recursion:** There is no depth limit or cycle detection. If a server redirects `A -> B -> A`, the method calls itself indefinitely until it raises a `RecursionError`.
* **Relative Redirect Failure:** The method does not resolve relative `Location` headers (e.g., `/login`). It passes the relative path directly to `urlparse(url).netloc`, which returns an empty string, causing the scraper to incorrectly mark valid URLs as invalid (`return None`).

**Expected Behavior**
* The scraper should stop following redirects after a reasonable limit (e.g., 30 hops) to prevent crashes.
* It should correctly resolve relative redirects (e.g., converting `/about` to `http://test.com/about`) using `urllib.parse.urljoin`.

**Proposed Fix**
I propose updating `verify_url` to:
* Accept a `depth` parameter to track recursion depth.
* Use `urljoin(url, location)` to normalize redirect URLs before the recursive call.

I have a fix ready and can submit a PR.

**Are you going to work on fixing this?**

- [X] Yes
- [ ] No

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.