aio-libs / aio-libs/yarl

Why does yarl encode query params differently?

Đang mở
#301 5 bình luận 1 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
1.5k
Fork
215
Merge trung bình
1 ngày 2 phút
Pull request đã merge (30 ngày)
13

Mô tả

**Note:** I'm not filing this issue to say "yarl is wrong, urllib is right"---I know that the author of yarl knows RFC 3986 way, way better than I ever will and it could be yarl that is doing the better job of adhering to it. But I'm curious what motivates the difference in behavior below.

**Summary:** We have an API endpoint where one of the query string params is `url=`. `aiohttp` (via `yarl.URL`) and `requests` (via `urllib.parse.urlencode`) encode the query string differently.

```python
>>> from yarl import URL
>>> url = URL("http://api.example.com")
>>> params = {
... "url": "https://www.katherinetimes.com.au/story/6005621/wife-of-ex-nissan-boss-ghosn-leaves-japan/?src=rss",
... "api_key": "XXXX",
... }
>>> url.with_query(params)
URL('http://api.example.com/?url=https://www.katherinetimes.com.au/story/6005621/wife-of-ex-nissan-boss-ghosn-leaves-japan/?src%3Drss&api_key=XXXX')
```

Notice that the **?** in the URL (within the query string) is *not* encoded:

```
... japan/?src%3Drss&api ...
```

With the `?` not encoded, the API endpoint gives us back a 400 response.

Now with `urllib.parse`:

```python
>>> parts = urllib.parse.urlparse(url.human_repr())
>>> parts
ParseResult(scheme='http', netloc='api.example.com', path='/', params='', query='', fragment='')
>>> urllib.parse.urlunparse(
... (
... parts.scheme,
... parts.netloc,
... parts.path,
... parts.params,
... urllib.parse.urlencode(params),
... parts.fragment
... )
... )
'http://api.example.com/?url=https%3A%2F%2Fwww.katherinetimes.com.au%2Fstory%2F6005621%2Fwife-of-ex-nissan-boss-ghosn-leaves-japan%2F%3Fsrc%3Drss&api_key=XXXX'
```

Here the **?** in the URL (within the query string) is encoded (as are the `/`).

```
... japan%2F%3Fsrc%3Drss&api ...
```

With the `?` encoded, the API endpoint gives us back a 200 response.

So, `urlencode` has a default of `safe=''` and encodes pretty much everything. Is there a reason that `yarl` seems to diverge from this?

Now, when I look at RFC 3986, it actually seems like the `/` and `?` should *not* need to be encoded:

```
query = *( pchar / "/" / "?" )
```

That is ironic, that it seems like yarl gets it "right" by leaving them unencoded, while urllib does too much work but then get us a 200 response.

So to make my question concrete:

- Are my assumptions/understanding here correct?
- Have you seen other situations like this?
- Within `aiohttp`, is there optionality to specify something like `safe`, or a callable that is called to the actual encoding of `params`?

The problem here is that even if `yarl` is "right," it basically forces the encoding even if I pass the full raw URL string to something like `aiohttp.request`:

```python
>>> URL('http://api.example.com?url=https%3A%2F%2Fwww.katherinetimes.com.au%2Fstory%2F6005621%2Fwife-of-ex-nissan-boss-ghosn-leaves-japan%2F%3Fsrc%3Drss&api_key=xxx')
URL('http://api.example.com/?url=https://www.katherinetimes.com.au/story/6005621/wife-of-ex-nissan-boss-ghosn-leaves-japan/?src%3Drss&api_key=xxx')
```

```python
>>> u = URL('http://api.example.com?url=https%3A%2F%2Fwww.katherinetimes.com.au%2Fstory%2F6005621%2Fwife-of-ex-nissan-boss-ghosn-leaves-japan%2F%3Fsrc%3Drss&api_key=xxx')
>>> u.raw_query_string
'url=https://www.katherinetimes.com.au/story/6005621/wife-of-ex-nissan-boss-ghosn-leaves-japan/?src%3Drss&api_key=xxx'
```

In `aiohttp`, the question then becomes "how can I override this behavior?" This is the answer I can come up with:

```python
async def fullencode(params, *, _bp=BASE_PARTS):
return URL(urllib.parse.urlunparse((
_bp.scheme, _bp.netloc, _bp.path, _bp.params,
urllib.parse.urlencode(params),
_bp.fragment
)), encoded=True)
```

Then pass `await fullencode(params)` to `request()`. (This assumes you have some base URL that forms `_bp` above.)

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.