Duke-GCB / Duke-GCB/bespin-api
Using habanero for content negotiation can lose character encoding
- Dominant language
- Python
- Stars
- 0
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
I suspect the severity here may depend on the server that resolves the DOI ultimately, but for the citations we're fetching via habanero, some of the characters are getting mangled due to incorrect encoding.
See https://gist.github.com/dleehr/006ddf61f982e56922c5642edf57a1a9
Looking up doi:10.1093/bioinformatics/btp324 results in different output from `habanero.cn` vs a standard `curl`
lookup_curl.sh:
```
...Bioinformatics, 25(14), 1754–1760. doi:10.1093/bioinformatics/btp324
```
lookup_habanero.py:
```
...Bioinformatics, 25(14), 1754â1760. doi:10.1093/bioinformatics/btp324
```
`1754–1760` vs `1754â1760` Note that this character is an [en-dash](http://www.fileformat.info/info/unicode/char/2013/index.htm) (–), not a hyphen (-)
```
>>> '–'.encode('utf-8')
b'\xe2\x80\x93'
>>> '-'.encode('utf-8')
b'-
>>> 'â'.encode('utf-8')
b'\xc3\xa2\xc2\x80\xc2\x93'
```
When `requests` gets the response inside `cn`, it marks the encoding as `ISO-8859-1`, which appears to be correct. However, requests returns the response text without the encoding, so unless you know what to expect, you can't decode it properly.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.