Using a `json.JSONDecoder` fails when simplejson is present
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 54.3k
- Forks
- 10.4k
- Avg merge
- 16h 43m
- Merged PRs (30d)
- 3
Description
It appears that when simplejson is in the environment requests preferentially imports it. However, the arguments accepted by JSONDecoder are inconsistent between the standard library json and simplejson leading to errors when using the standard json.JSONDecoder when simplejson is in the environment.
Your documentation for response.json() says:
Parameters: | **kwargs – Optional arguments that json.loads takes.
The documentation for json.loads says:
json.loads(s, *, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw)
and
To use a custom JSONDecoder subclass, specify it with the cls kwarg; otherwise JSONDecoder is used. Additional keyword arguments will be passed to the constructor of the class.
I expected to be able to use a custom json.JSONDecoder without issue and indeed this works on Python 2.7, but in Python 3.7 it fails.
I can see that the issue of preferentially importing simplejson has been raised a few times e.g.
https://github.com/requests/requests/pull/2516
https://github.com/requests/requests/issues/3052
and that it is slated for removal in Requests 3, which I guess will resolve this issue.
If it is not possible to resolve this issue some other way in Requests 2.x then it would be nice if the documentation around response.json() was updated to make it clear that the arguments to json.loads could be either simplejson.loads or the standard lib json.loads depending on the environment since the preferential import is effectively undocumented and non-obvious.
Expected Result
A successful JSON load when using r.json(cls=json.JSONDecoder).
Actual Result
print(r.json(cls=json.JSONDecoder))
File ".../python3.7/site-packages/requests/models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File ".../python3.7/site-packages/simplejson/__init__.py", line 535, in loads
return cls(encoding=encoding, **kw).decode(s)
TypeError: __init__() got an unexpected keyword argument 'encoding'
Reproduction Steps
pip install requests
pip install simplejson
import json
import requests
r = requests.get('http://localhost:5984')
print(r.json(cls=json.JSONDecoder))
System Information
Note this issue does not occur in Python 2.7.x.
$ python -m requests.help
{
"chardet": {
"version": "3.0.4"
},
"cryptography": {
"version": ""
},
"idna": {
"version": "2.7"
},
"implementation": {
"name": "CPython",
"version": "3.7.0"
},
"platform": {
"release": "18.0.0",
"system": "Darwin"
},
"pyOpenSSL": {
"openssl_version": "",
"version": null
},
"requests": {
"version": "2.20.0"
},
"system_ssl": {
"version": "1000210f"
},
"urllib3": {
"version": "1.24"
},
"using_pyopenssl": false
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in requests/models.py at Response.json(), where the traceback shows complexjson.loads is called, and compare that behavior with the documented response.json() and json.loads APIs. Reproduce the failure with simplejson installed and a custom json.JSONDecoder; done means the reported call succeeds or the response.json() documentation clearly describes the environment-dependent arguments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100