aio-libs / aio-libs/aiohttp

Wrong URL parsing in redirect

Offen
#6,626 12 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug
Vorherrschende Sprache
Python
Sterne
16.5k
Forks
2.4k
Ø Merge
17 Std. 22 Min.
Gemergte PRs (30 T.)
212

Beschreibung

### Describe the bug

I'm scrapping the marvel fandom site using tasks of asyncio.
Every time that I try to get the HTML document from a URL that contains the '&' symbol I have the same issue:

For example with: https://marvel.fandom.com/wiki/Amazing_Spider-Man_&_Silk:_The_Spider(fly)_Effect_Vol_1

This link redirects to another link where the & symbol is replaced with the % encoding (& = %26). So the actual URL of the page is https://marvel.fandom.com/wiki/Amazing_Spider-Man_%26_Silk:_The_Spider(fly)_Effect_Vol_1

Then I run the following code and I get an error

![image](https://user-images.githubusercontent.com/18238415/154723378-0ec00927-6a4a-4a32-9350-9c98241a6337.png)

To catch the exception I made the following:

![image](https://user-images.githubusercontent.com/18238415/154724157-fa51c738-f2c1-4bea-ac90-460443c36a34.png)

And as you can see the code redirects many times until max_amount_redirects is reached. The problem is that the actual URL obtained in the redirection ('Location': 'https://marvel.fandom.com/wiki/Amazing_Spider-Man_%26_Silk:_The_Spider(fly)_Effect_Vol_1') apparently is again parsed with % code and %26 is replaced by &.

The same error occurs if I use https://marvel.fandom.com/wiki/Amazing_Spider-Man_%26_Silk:_The_Spider(fly)_Effect_Vol_1 in the first time, %26 is replaced by & and redirects until limit is reached. To make it work I had to replace & with %%26 so it parses to a literal "%26" in the url.

### To Reproduce

![image](https://user-images.githubusercontent.com/18238415/154723378-0ec00927-6a4a-4a32-9350-9c98241a6337.png)

### Expected behavior

Redirects correctly

### Logs/tracebacks

```python-traceback
No applicable
```

### Python Version

```console
$ python --version

Python 3.10.1
```

### aiohttp Version

```console
$ python -m pip show aiohttp

Name: aiohttp
Version: 3.8.1
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: None
Author-email: None
License: Apache 2
```

### multidict Version

```console
$ python -m pip show multidict

Name: multidict
Version: 6.0.2
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
```

### yarl Version

```console
$ python -m pip show yarl

Name: yarl
Version: 1.7.2
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
```

### OS

Manjaro, Linux

### Related component

Client

### Additional context

_No response_

### Code of Conduct

- [X] I agree to follow the aio-libs Code of Conduct

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.