content-type: text/csv does not apply UTF-8-decoding by default (RFC 7111 violation?)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 54.3k
- Forks
- 10.4k
- Avg merge
- 16h 43m
- Merged PRs (30d)
- 3
Description
Created a test/repro/MWE sheet here with a cell containing the letter ö: https://docs.google.com/spreadsheets/d/1q02F0AjDfCo_XlgFtT7HHU96emOcyf0PRQGKGoQTXCE/edit?usp=sharing
When you HTTP GET https://docs.google.com/spreadsheets/d/1q02F0AjDfCo_XlgFtT7HHU96emOcyf0PRQGKGoQTXCE/export?format=csv Google generates and sends a CSV document with UTF-8-encoded text in the response body.
Fetching this URL with requests and accessing the response content with the text attribute reveals that requests does not UTF-8-decode the response body bytes:
$ python -c 'import requests; r=requests.get("https://docs.google.com/spreadsheets/d/1q02F0AjDfCo_XlgFtT7HHU96emOcyf0PRQGKGoQTXCE/export?format=csv"); print(r.text)'
foo,bar,umlaut: öö
Library version:
$ pip list | grep requests
requests 2.25.1
Response headers (from a curl -v ...):
> GET /<snip> HTTP/2
> Host: doc-14-2s-sheets.googleusercontent.com
> user-agent: curl/7.69.1
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [264 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 200
< content-type: text/csv
< x-robots-tag: noindex, nofollow, nosnippet
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: Mon, 01 Jan 1990 00:00:00 GMT
< date: Wed, 10 Feb 2021 10:12:52 GMT
< content-disposition: attachment; filename="<snip>.csv"; filename*=UTF-8''<snip>.csv
< access-control-allow-origin: *
< access-control-expose-headers: Cache-Control,Content-Disposition,Content-Encoding,Content-Length,Content-Type,Date,Expires,Pragma,Server,Transfer-Encoding
< content-security-policy: base-uri 'self';object-src 'self';report-uri https://docs.google.com/spreadsheets/cspreport;script-src 'nonce-+Qyt<snip>w' 'unsafe-inline' 'strict-dynamic' https: http: 'unsafe-eval';worker-src 'self'
< content-security-policy: frame-ancestors 'self' https://docs.google.com
< x-frame-options: ALLOW-FROM https://docs.google.com
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< server: GSE
< alt-svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
< accept-ranges: none
< vary: Accept-Encoding
So, there is content-type: text/csv, i.e. no 'charset' specification.
I am not sure if RFC 7111 is the most recent / authoritative reference, but it says about the text/csv media type that
The "charset" parameter specifies the charset employed by the CSV
content. In accordance with RFC 6657 [RFC6657], the charset
parameter SHOULD be used, and if it is not present, UTF-8 SHOULD
be assumed as the default (this implies that US-ASCII CSV will
work, even when not specifying the "charset" parameter). Any
charset defined by IANA for the "text" tree may be used in
conjunction with the "charset" parameter.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the behavior with the documented requests.get call and inspect how response.text handles a text/csv response without a charset. Compare the result with RFC 7111's UTF-8 default; done when the reported CSV response decodes as expected and the behavior has regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100