aio-libs / aio-libs/yarl

Why does yarl encode query params differently?

未關閉
#301 5 則留言 1 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
1.5k
分支
215
平均合併
1 天 2 分鐘
30 天內合併 PR
13

描述

**Note:** I'm not filing this issue to say "yarl is wrong, urllib is right"---I know that the author of yarl knows RFC 3986 way, way better than I ever will and it could be yarl that is doing the better job of adhering to it. But I'm curious what motivates the difference in behavior below.

**Summary:** We have an API endpoint where one of the query string params is `url=`. `aiohttp` (via `yarl.URL`) and `requests` (via `urllib.parse.urlencode`) encode the query string differently.

```python
>>> from yarl import URL
>>> url = URL("http://api.example.com")
>>> params = {
... "url": "https://www.katherinetimes.com.au/story/6005621/wife-of-ex-nissan-boss-ghosn-leaves-japan/?src=rss",
... "api_key": "XXXX",
... }
>>> url.with_query(params)
URL('http://api.example.com/?url=https://www.katherinetimes.com.au/story/6005621/wife-of-ex-nissan-boss-ghosn-leaves-japan/?src%3Drss&api_key=XXXX')
```

Notice that the **?** in the URL (within the query string) is *not* encoded:

```
... japan/?src%3Drss&api ...
```

With the `?` not encoded, the API endpoint gives us back a 400 response.

Now with `urllib.parse`:

```python
>>> parts = urllib.parse.urlparse(url.human_repr())
>>> parts
ParseResult(scheme='http', netloc='api.example.com', path='/', params='', query='', fragment='')
>>> urllib.parse.urlunparse(
... (
... parts.scheme,
... parts.netloc,
... parts.path,
... parts.params,
... urllib.parse.urlencode(params),
... parts.fragment
... )
... )
'http://api.example.com/?url=https%3A%2F%2Fwww.katherinetimes.com.au%2Fstory%2F6005621%2Fwife-of-ex-nissan-boss-ghosn-leaves-japan%2F%3Fsrc%3Drss&api_key=XXXX'
```

Here the **?** in the URL (within the query string) is encoded (as are the `/`).

```
... japan%2F%3Fsrc%3Drss&api ...
```

With the `?` encoded, the API endpoint gives us back a 200 response.

So, `urlencode` has a default of `safe=''` and encodes pretty much everything. Is there a reason that `yarl` seems to diverge from this?

Now, when I look at RFC 3986, it actually seems like the `/` and `?` should *not* need to be encoded:

```
query = *( pchar / "/" / "?" )
```

That is ironic, that it seems like yarl gets it "right" by leaving them unencoded, while urllib does too much work but then get us a 200 response.

So to make my question concrete:

- Are my assumptions/understanding here correct?
- Have you seen other situations like this?
- Within `aiohttp`, is there optionality to specify something like `safe`, or a callable that is called to the actual encoding of `params`?

The problem here is that even if `yarl` is "right," it basically forces the encoding even if I pass the full raw URL string to something like `aiohttp.request`:

```python
>>> URL('http://api.example.com?url=https%3A%2F%2Fwww.katherinetimes.com.au%2Fstory%2F6005621%2Fwife-of-ex-nissan-boss-ghosn-leaves-japan%2F%3Fsrc%3Drss&api_key=xxx')
URL('http://api.example.com/?url=https://www.katherinetimes.com.au/story/6005621/wife-of-ex-nissan-boss-ghosn-leaves-japan/?src%3Drss&api_key=xxx')
```

```python
>>> u = URL('http://api.example.com?url=https%3A%2F%2Fwww.katherinetimes.com.au%2Fstory%2F6005621%2Fwife-of-ex-nissan-boss-ghosn-leaves-japan%2F%3Fsrc%3Drss&api_key=xxx')
>>> u.raw_query_string
'url=https://www.katherinetimes.com.au/story/6005621/wife-of-ex-nissan-boss-ghosn-leaves-japan/?src%3Drss&api_key=xxx'
```

In `aiohttp`, the question then becomes "how can I override this behavior?" This is the answer I can come up with:

```python
async def fullencode(params, *, _bp=BASE_PARTS):
return URL(urllib.parse.urlunparse((
_bp.scheme, _bp.netloc, _bp.path, _bp.params,
urllib.parse.urlencode(params),
_bp.fragment
)), encoded=True)
```

Then pass `await fullencode(params)` to `request()`. (This assumes you have some base URL that forms `_bp` above.)

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。