aio-libs / aio-libs/aiohttp

Wrong URL parsing in redirect

Abierto
#6,626 12 comentarios 0 reacciones 0 asignados Ver en GitHub
bug
Lenguaje dominante
Python
Estrellas
16.5k
Forks
2.4k
Merge medio
17 h 22 min
PR fusionados (30 d)
212

Descripción

### Describe the bug

I'm scrapping the marvel fandom site using tasks of asyncio.
Every time that I try to get the HTML document from a URL that contains the '&' symbol I have the same issue:

For example with: https://marvel.fandom.com/wiki/Amazing_Spider-Man_&_Silk:_The_Spider(fly)_Effect_Vol_1

This link redirects to another link where the & symbol is replaced with the % encoding (& = %26). So the actual URL of the page is https://marvel.fandom.com/wiki/Amazing_Spider-Man_%26_Silk:_The_Spider(fly)_Effect_Vol_1

Then I run the following code and I get an error

![image](https://user-images.githubusercontent.com/18238415/154723378-0ec00927-6a4a-4a32-9350-9c98241a6337.png)

To catch the exception I made the following:

![image](https://user-images.githubusercontent.com/18238415/154724157-fa51c738-f2c1-4bea-ac90-460443c36a34.png)

And as you can see the code redirects many times until max_amount_redirects is reached. The problem is that the actual URL obtained in the redirection ('Location': 'https://marvel.fandom.com/wiki/Amazing_Spider-Man_%26_Silk:_The_Spider(fly)_Effect_Vol_1') apparently is again parsed with % code and %26 is replaced by &.

The same error occurs if I use https://marvel.fandom.com/wiki/Amazing_Spider-Man_%26_Silk:_The_Spider(fly)_Effect_Vol_1 in the first time, %26 is replaced by & and redirects until limit is reached. To make it work I had to replace & with %%26 so it parses to a literal "%26" in the url.

### To Reproduce

![image](https://user-images.githubusercontent.com/18238415/154723378-0ec00927-6a4a-4a32-9350-9c98241a6337.png)

### Expected behavior

Redirects correctly

### Logs/tracebacks

```python-traceback
No applicable
```

### Python Version

```console
$ python --version

Python 3.10.1
```

### aiohttp Version

```console
$ python -m pip show aiohttp

Name: aiohttp
Version: 3.8.1
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: None
Author-email: None
License: Apache 2
```

### multidict Version

```console
$ python -m pip show multidict

Name: multidict
Version: 6.0.2
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
```

### yarl Version

```console
$ python -m pip show yarl

Name: yarl
Version: 1.7.2
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
```

### OS

Manjaro, Linux

### Related component

Client

### Additional context

_No response_

### Code of Conduct

- [X] I agree to follow the aio-libs Code of Conduct

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.