Pyarrow allows writing arbitrary bytestrings in Parquet metadata but spec only allows UTF-8
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I'm not sure whether the Parquet spec is overly restrictive or pyarrow overly permissive, but pyarrow allows writing arbitrary bytestrings as metadata. But in Parquet the metadata is a `string:string` mapping. A modified extension type example which will write illegal bytes into the Parquet metadata:
```python
import pyarrow as pa
import pyarrow.parquet as pq
class PeriodType(pa.ExtensionType):
def __init__(self, freq):
self._freq = freq
super().__init__(pa.int64(), "my_package.period")
@property
def freq(self):
return self._freq
def __arrow_ext_serialize__(self):
return b"\xff" + "freq={}".format(self.freq).encode() + b"\xff"
@classmethod
def __arrow_ext_deserialize__(cls, storage_type, serialized):
print("deserializing:", serialized)
assert serialized[0] == 0xff and serialized[-1] == 0xff
serialized = serialized[1:-1].decode()
assert serialized.startswith("freq=")
freq = serialized.split("=")[1]
return PeriodType(freq)
pa.register_extension_type(PeriodType(0.0))
storage = pa.array([1, 2, 3], pa.int64())
arr = pa.ExtensionArray.from_storage(PeriodType(0.0), storage)
tab = pa.Table.from_arrays([arr], names=["periods"])
pq.write_table(tab, 'example.parquet')
print(pq.read_table('example.parquet').schema)
```
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.