"UnicodeEncodeError: 'latin-1' codec can't encode character, ordinal not in range" on GCS upload_object_via_stream
- 主要语言
- Python
- 星标
- 2.1k
- 派生
- 931
- 平均合并
- 1 天 2 小时
- 30 天内合并 PR
- 4
描述
## Summary
I'm trying to upload some CSV files to GCS using libcloud, today I this error:
```log
UnicodeEncodeError: 'latin-1' codec can't encode character '\u201c' in position 628: ordinal not in range(256)
```
## Detailed Information
The encode character and position change with each run, even though the files don't change. I'm not sure what's causing that, CSV files are basically plaintext.
Here's the full stack (some internal code ommitted):
```log
Traceback (most recent call last):
[...]
self.__container.upload_object_via_stream(stream, name) # type: ignore
File "/usr/local/lib/python3.8/site-packages/libcloud/storage/base.py", line 227, in upload_object_via_stream
return self.driver.upload_object_via_stream(
File "/usr/local/lib/python3.8/site-packages/libcloud/storage/drivers/s3.py", line 843, in upload_object_via_stream
return self._put_object(
File "/usr/local/lib/python3.8/site-packages/libcloud/storage/drivers/s3.py", line 1007, in _put_object
result_dict = self._upload_object(
File "/usr/local/lib/python3.8/site-packages/libcloud/storage/base.py", line 874, in _upload_object
response = self.connection.request(
File "/usr/local/lib/python3.8/site-packages/libcloud/common/base.py", line 659, in request
return request_to_be_executed(
File "/usr/local/lib/python3.8/site-packages/libcloud/common/base.py", line 678, in _retryable_request
self.connection.prepared_request(
File "/usr/local/lib/python3.8/site-packages/libcloud/http.py", line 260, in prepared_request
self.response = self.session.send(
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/local/lib/python3.8/http/client.py", line 1256, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1302, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1251, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1040, in _send_output
for chunk in chunks:
File "/usr/local/lib/python3.8/http/client.py", line 999, in _read_readable
datablock = datablock.encode("iso-8859-1")
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2019' in position 5251: ordinal not in range(256)
```
I'm running this from inside a k8s pod in a GKE cluster, trying to write to a GCS bucket.
```
apache-libcloud version: 3.5.1
python version: 3.8.15
pod image: python:3.8-slim
```
I'm not sure what are the steps to reproduce it. It's a CSV file generated by pulling some BigQuery schema metadata. It works locally, but when I ran it in the pod it raised this error.
贡献指南
调研方向
从 libcloud/storage/base.py 中的 upload_object_via_stream 和 libcloud/storage/drivers/s3.py 中的 S3 驱动路径开始,然后跟踪请求通过 libcloud/common/base.py 和 libcloud/http.py 的准备过程。在报告中的 Python 和容器环境中,使用包含非 Latin-1 字符的 CSV 重现该失败。完成标准是上传能够处理此类 CSV 内容而不出现 UnicodeEncodeError,并为失败路径包含回归覆盖。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- google-cloud, python
- 领域
- cloud
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 32/100