aio-libs / aio-libs/aiohttp

Wrong URL parsing in redirect

オープン
#6,626 コメント 12 件 リアクション 0 件 担当者 0 名 GitHub で見る
bug
主要言語
Python
スター
16.5k
フォーク
2.4k
平均マージ
17時間 22分
マージ済み PR(30日)
212

説明

### Describe the bug

I'm scrapping the marvel fandom site using tasks of asyncio.
Every time that I try to get the HTML document from a URL that contains the '&' symbol I have the same issue:

For example with: https://marvel.fandom.com/wiki/Amazing_Spider-Man_&_Silk:_The_Spider(fly)_Effect_Vol_1

This link redirects to another link where the & symbol is replaced with the % encoding (& = %26). So the actual URL of the page is https://marvel.fandom.com/wiki/Amazing_Spider-Man_%26_Silk:_The_Spider(fly)_Effect_Vol_1

Then I run the following code and I get an error

![image](https://user-images.githubusercontent.com/18238415/154723378-0ec00927-6a4a-4a32-9350-9c98241a6337.png)

To catch the exception I made the following:

![image](https://user-images.githubusercontent.com/18238415/154724157-fa51c738-f2c1-4bea-ac90-460443c36a34.png)

And as you can see the code redirects many times until max_amount_redirects is reached. The problem is that the actual URL obtained in the redirection ('Location': 'https://marvel.fandom.com/wiki/Amazing_Spider-Man_%26_Silk:_The_Spider(fly)_Effect_Vol_1') apparently is again parsed with % code and %26 is replaced by &.

The same error occurs if I use https://marvel.fandom.com/wiki/Amazing_Spider-Man_%26_Silk:_The_Spider(fly)_Effect_Vol_1 in the first time, %26 is replaced by & and redirects until limit is reached. To make it work I had to replace & with %%26 so it parses to a literal "%26" in the url.

### To Reproduce

![image](https://user-images.githubusercontent.com/18238415/154723378-0ec00927-6a4a-4a32-9350-9c98241a6337.png)

### Expected behavior

Redirects correctly

### Logs/tracebacks

```python-traceback
No applicable
```

### Python Version

```console
$ python --version

Python 3.10.1
```

### aiohttp Version

```console
$ python -m pip show aiohttp

Name: aiohttp
Version: 3.8.1
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: None
Author-email: None
License: Apache 2
```

### multidict Version

```console
$ python -m pip show multidict

Name: multidict
Version: 6.0.2
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
```

### yarl Version

```console
$ python -m pip show yarl

Name: yarl
Version: 1.7.2
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
```

### OS

Manjaro, Linux

### Related component

Client

### Additional context

_No response_

### Code of Conduct

- [X] I agree to follow the aio-libs Code of Conduct

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。