apache / apache/arrow

Pyarrow allows writing arbitrary bytestrings in Parquet metadata but spec only allows UTF-8

Open
#47,893 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I'm not sure whether the Parquet spec is overly restrictive or pyarrow overly permissive, but pyarrow allows writing arbitrary bytestrings as metadata. But in Parquet the metadata is a `string:string` mapping. A modified extension type example which will write illegal bytes into the Parquet metadata:

```python
import pyarrow as pa
import pyarrow.parquet as pq

class PeriodType(pa.ExtensionType):
def __init__(self, freq):
self._freq = freq
super().__init__(pa.int64(), "my_package.period")

@property
def freq(self):
return self._freq

def __arrow_ext_serialize__(self):
return b"\xff" + "freq={}".format(self.freq).encode() + b"\xff"

@classmethod
def __arrow_ext_deserialize__(cls, storage_type, serialized):
print("deserializing:", serialized)
assert serialized[0] == 0xff and serialized[-1] == 0xff
serialized = serialized[1:-1].decode()
assert serialized.startswith("freq=")
freq = serialized.split("=")[1]
return PeriodType(freq)

pa.register_extension_type(PeriodType(0.0))

storage = pa.array([1, 2, 3], pa.int64())
arr = pa.ExtensionArray.from_storage(PeriodType(0.0), storage)

tab = pa.Table.from_arrays([arr], names=["periods"])
pq.write_table(tab, 'example.parquet')
print(pq.read_table('example.parquet').schema)
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.