apache / apache/arrow

[C++][Parquet] Extension types with nanosecond timestamp resolution don't roundtrip

Open
#31,163 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Example code:

```python

import pyarrow as pa
import pyarrow.parquet as pq

class MyTimestampType(pa.PyExtensionType):

def __init__(self):
pa.PyExtensionType.__init__(self, pa.timestamp("ns"))

def __reduce__(self):
return MyTimestampType, ()

arr = MyTimestampType().wrap_array(pa.array([1000, 2000, 3000], pa.timestamp("ns")))
table = pa.table({"col": arr})
```

```Java

>>> table.schema
col: extension>

>>> pq.write_table(table, "test_parquet_extension_type_timestamp_ns.parquet")
>>> result = pq.read_table("test_parquet_extension_type_timestamp_ns.parquet")
>>> result.schema
col: timestamp[us]
```

The reason for this is because we only restore the extension type if the inferred storage type (inferred from parquet + after applying any updates based on the Arrow schema) exactly equals the original storage type (as stored in the Arrow schema):

https://github.com/apache/arrow/blob/afaa92e7e4289d6e4f302cc91810368794e8092b/cpp/src/parquet/arrow/schema.cc#L973-L977

And, with the default options, a timestamp with nanosecond resolution gets stored as microsecond resolution in Parquet, and that is something we do not restore when updating the read types based on the stored Arrow schema (eg we do add a timezone, but we don't change the resolution).

An additional issue is that _if_ you loose the extension type, the field metadata about the extension type are also lost. I think that if we cannot restore the extension type, we should at least try to keep the ARROW:extension field metadata as information. This is also what we do for an unrecognized (unregistered) extension type.

**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-15711) / @jorisvandenbossche

**Note**: *This issue was originally created as [ARROW-15711](https://issues.apache.org/jira/browse/ARROW-15711). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the Python example with pyarrow and parquet, then inspect cpp/src/parquet/arrow/schema.cc around lines 973-977, where extension restoration depends on the inferred and original storage types matching. Done means nanosecond timestamp extension types roundtrip when possible, while ARROW:extension metadata are retained when restoration is not possible.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.