psf / psf/requests

content-type: text/csv does not apply UTF-8-decoding by default (RFC 7111 violation?)

Open
#5,746 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.4k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

Created a test/repro/MWE sheet here with a cell containing the letter ö: https://docs.google.com/spreadsheets/d/1q02F0AjDfCo_XlgFtT7HHU96emOcyf0PRQGKGoQTXCE/edit?usp=sharing

When you HTTP GET https://docs.google.com/spreadsheets/d/1q02F0AjDfCo_XlgFtT7HHU96emOcyf0PRQGKGoQTXCE/export?format=csv Google generates and sends a CSV document with UTF-8-encoded text in the response body.

Fetching this URL with requests and accessing the response content with the text attribute reveals that requests does not UTF-8-decode the response body bytes:

$ python -c 'import requests; r=requests.get("https://docs.google.com/spreadsheets/d/1q02F0AjDfCo_XlgFtT7HHU96emOcyf0PRQGKGoQTXCE/export?format=csv"); print(r.text)'
foo,bar,umlaut: öö

Library version:

$ pip list | grep requests
requests                 2.25.1

Response headers (from a curl -v ...):

> GET /<snip> HTTP/2
> Host: doc-14-2s-sheets.googleusercontent.com
> user-agent: curl/7.69.1
> accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [264 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 200 
< content-type: text/csv
< x-robots-tag: noindex, nofollow, nosnippet
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: Mon, 01 Jan 1990 00:00:00 GMT
< date: Wed, 10 Feb 2021 10:12:52 GMT
< content-disposition: attachment; filename="<snip>.csv"; filename*=UTF-8''<snip>.csv
< access-control-allow-origin: *
< access-control-expose-headers: Cache-Control,Content-Disposition,Content-Encoding,Content-Length,Content-Type,Date,Expires,Pragma,Server,Transfer-Encoding
< content-security-policy: base-uri 'self';object-src 'self';report-uri https://docs.google.com/spreadsheets/cspreport;script-src 'nonce-+Qyt<snip>w' 'unsafe-inline' 'strict-dynamic' https: http: 'unsafe-eval';worker-src 'self'
< content-security-policy: frame-ancestors 'self' https://docs.google.com
< x-frame-options: ALLOW-FROM https://docs.google.com
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< server: GSE
< alt-svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
< accept-ranges: none
< vary: Accept-Encoding

So, there is content-type: text/csv, i.e. no 'charset' specification.

I am not sure if RFC 7111 is the most recent / authoritative reference, but it says about the text/csv media type that

      The "charset" parameter specifies the charset employed by the CSV
      content.  In accordance with RFC 6657 [RFC6657], the charset
      parameter SHOULD be used, and if it is not present, UTF-8 SHOULD
      be assumed as the default (this implies that US-ASCII CSV will
      work, even when not specifying the "charset" parameter).  Any
      charset defined by IANA for the "text" tree may be used in
      conjunction with the "charset" parameter.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the behavior with the documented requests.get call and inspect how response.text handles a text/csv response without a charset. Compare the result with RFC 7111's UTF-8 default; done when the reported CSV response decodes as expected and the behavior has regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.