Duke-GCB / Duke-GCB/bespin-api

Using habanero for content negotiation can lose character encoding

Open
#112 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
2
PR merge metrics
No merged PRs in 30d

Description

I suspect the severity here may depend on the server that resolves the DOI ultimately, but for the citations we're fetching via habanero, some of the characters are getting mangled due to incorrect encoding.

See https://gist.github.com/dleehr/006ddf61f982e56922c5642edf57a1a9

Looking up doi:10.1093/bioinformatics/btp324 results in different output from `habanero.cn` vs a standard `curl`

lookup_curl.sh:
```
...Bioinformatics, 25(14), 1754–1760. doi:10.1093/bioinformatics/btp324
```

lookup_habanero.py:
```
...Bioinformatics, 25(14), 1754–1760. doi:10.1093/bioinformatics/btp324
```

`1754–1760` vs `1754–1760` Note that this character is an [en-dash](http://www.fileformat.info/info/unicode/char/2013/index.htm) (–), not a hyphen (-)

```
>>> '–'.encode('utf-8')
b'\xe2\x80\x93'
>>> '-'.encode('utf-8')
b'-
>>> '–'.encode('utf-8')
b'\xc3\xa2\xc2\x80\xc2\x93'
```

When `requests` gets the response inside `cn`, it marks the encoding as `ISO-8859-1`, which appears to be correct. However, requests returns the response text without the encoding, so unless you know what to expect, you can't decode it properly.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.