[Python] Non-UTF-8 bytes should be disallowed in custom_metadata
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
[Schema.fbs](https://github.com/apache/arrow/blob/main/format/Schema.fbs#L475) requires that custom metadata keys and values are strings, which in flatbuffer world are [length-prefixed, null-terminated runs of UTF-8 or 7-bit ASCII code points](https://flatbuffers.dev/schema/#strings). However, pyarrow allows you to create and read metadata with arbitrary bytes. This is inconsistent with other implementations, such as rust, which uses the String type, enforcing UTF-8.
### Component(s)
Python
### Example
```python
import pyarrow as pa
b = b"\xff\xfe\xfa"
try:
b.decode("utf-8")
except UnicodeDecodeError:
print("byte string is invalid UTF-8")
schema = pa.schema([pa.field("id", pa.uint32())], metadata={b: b})
table = pa.table({"id": [1, 2]}, schema=schema)
```
Contributor guide
Assessment
This issue has not been assessed yet.