Why does yarl encode query params differently?
- Ngôn ngữ chính
- Python
- Star
- 1.5k
- Fork
- 215
- Merge trung bình
- 1 ngày 2 phút
- Pull request đã merge (30 ngày)
- 13
Mô tả
**Note:** I'm not filing this issue to say "yarl is wrong, urllib is right"---I know that the author of yarl knows RFC 3986 way, way better than I ever will and it could be yarl that is doing the better job of adhering to it. But I'm curious what motivates the difference in behavior below.
**Summary:** We have an API endpoint where one of the query string params is `url=`. `aiohttp` (via `yarl.URL`) and `requests` (via `urllib.parse.urlencode`) encode the query string differently.
```python
>>> from yarl import URL
>>> url = URL("http://api.example.com")
>>> params = {
... "url": "https://www.katherinetimes.com.au/story/6005621/wife-of-ex-nissan-boss-ghosn-leaves-japan/?src=rss",
... "api_key": "XXXX",
... }
>>> url.with_query(params)
URL('http://api.example.com/?url=https://www.katherinetimes.com.au/story/6005621/wife-of-ex-nissan-boss-ghosn-leaves-japan/?src%3Drss&api_key=XXXX')
```
Notice that the **?** in the URL (within the query string) is *not* encoded:
```
... japan/?src%3Drss&api ...
```
With the `?` not encoded, the API endpoint gives us back a 400 response.
Now with `urllib.parse`:
```python
>>> parts = urllib.parse.urlparse(url.human_repr())
>>> parts
ParseResult(scheme='http', netloc='api.example.com', path='/', params='', query='', fragment='')
>>> urllib.parse.urlunparse(
... (
... parts.scheme,
... parts.netloc,
... parts.path,
... parts.params,
... urllib.parse.urlencode(params),
... parts.fragment
... )
... )
'http://api.example.com/?url=https%3A%2F%2Fwww.katherinetimes.com.au%2Fstory%2F6005621%2Fwife-of-ex-nissan-boss-ghosn-leaves-japan%2F%3Fsrc%3Drss&api_key=XXXX'
```
Here the **?** in the URL (within the query string) is encoded (as are the `/`).
```
... japan%2F%3Fsrc%3Drss&api ...
```
With the `?` encoded, the API endpoint gives us back a 200 response.
So, `urlencode` has a default of `safe=''` and encodes pretty much everything. Is there a reason that `yarl` seems to diverge from this?
Now, when I look at RFC 3986, it actually seems like the `/` and `?` should *not* need to be encoded:
```
query = *( pchar / "/" / "?" )
```
That is ironic, that it seems like yarl gets it "right" by leaving them unencoded, while urllib does too much work but then get us a 200 response.
So to make my question concrete:
- Are my assumptions/understanding here correct?
- Have you seen other situations like this?
- Within `aiohttp`, is there optionality to specify something like `safe`, or a callable that is called to the actual encoding of `params`?
The problem here is that even if `yarl` is "right," it basically forces the encoding even if I pass the full raw URL string to something like `aiohttp.request`:
```python
>>> URL('http://api.example.com?url=https%3A%2F%2Fwww.katherinetimes.com.au%2Fstory%2F6005621%2Fwife-of-ex-nissan-boss-ghosn-leaves-japan%2F%3Fsrc%3Drss&api_key=xxx')
URL('http://api.example.com/?url=https://www.katherinetimes.com.au/story/6005621/wife-of-ex-nissan-boss-ghosn-leaves-japan/?src%3Drss&api_key=xxx')
```
```python
>>> u = URL('http://api.example.com?url=https%3A%2F%2Fwww.katherinetimes.com.au%2Fstory%2F6005621%2Fwife-of-ex-nissan-boss-ghosn-leaves-japan%2F%3Fsrc%3Drss&api_key=xxx')
>>> u.raw_query_string
'url=https://www.katherinetimes.com.au/story/6005621/wife-of-ex-nissan-boss-ghosn-leaves-japan/?src%3Drss&api_key=xxx'
```
In `aiohttp`, the question then becomes "how can I override this behavior?" This is the answer I can come up with:
```python
async def fullencode(params, *, _bp=BASE_PARTS):
return URL(urllib.parse.urlunparse((
_bp.scheme, _bp.netloc, _bp.path, _bp.params,
urllib.parse.urlencode(params),
_bp.fragment
)), encoded=True)
```
Then pass `await fullencode(params)` to `request()`. (This assumes you have some base URL that forms `_bp` above.)
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.