apache / apache/arrow

[Python] Non-UTF-8 bytes should be disallowed in custom_metadata

Open
#49,058 8 comments 0 reactions 1 assignee Claimed by @nitrajen View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

[Schema.fbs](https://github.com/apache/arrow/blob/main/format/Schema.fbs#L475) requires that custom metadata keys and values are strings, which in flatbuffer world are [length-prefixed, null-terminated runs of UTF-8 or 7-bit ASCII code points](https://flatbuffers.dev/schema/#strings). However, pyarrow allows you to create and read metadata with arbitrary bytes. This is inconsistent with other implementations, such as rust, which uses the String type, enforcing UTF-8.

### Component(s)

Python

### Example

```python
import pyarrow as pa

b = b"\xff\xfe\xfa"

try:
b.decode("utf-8")
except UnicodeDecodeError:
print("byte string is invalid UTF-8")

schema = pa.schema([pa.field("id", pa.uint32())], metadata={b: b})
table = pa.table({"id": [1, 2]}, schema=schema)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.