apache / apache/arrow

[C++] date64[ms] comes back as date32[day] after roundtrip to Parquet

Open
#15,032 1 comment 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I'm trying to build a tool to confirm that a given parquet file conforms to an expected schema with [our metadata format](https://github.com/moj-analytical-services/mojap-metadata). There appears to be a bug with how pyarrow converts dates, however.

I have a dummy dataset, called test.csv (expand to view)

|my_int|animal |my_email |my_datetime |my_date |
|------|-------|-------------------------|-------------------|----------|
|16 |NA |omar65@waller.info |2013-01-14 15:54:20|1993-09-27|
|13 |cat |nataliechavez@hotmail.com|2006-12-16 01:44:21|1989-04-08|
|13 |dog |stephanie12@leblanc.com |1972-06-19 11:22:59|2006-10-01|
|18 |NA |cameronscott@bradshaw.com|2009-07-11 00:45:28|1992-07-24|
|10 |fish |rossrebecca@gould.net |1970-09-11 22:26:13|2011-11-30|
|13 |cat |escobarryan@gmail.com |2018-09-18 16:18:37|1979-01-22|
|16 |dog |shelly51@mitchell.com |2002-08-26 10:25:03|1981-03-27|
|11 |dog |kwilliams@hotmail.com |1989-02-26 16:33:37|2012-06-19|
|19 |chicken|lisacharles@kennedy.com |1974-07-25 04:41:22|2003-11-06|
|11 |chicken|olsonchelsea@gray.org |1992-04-03 09:47:30|1972-07-06|

I have a user-generated schema that resolves to this:
```
my_int: int64
animal: string
my_email: string
my_datetime: timestamp[s]
my_date: date64[ms]
```

However, the last column seems to be read by pyarrow as `date32[day]`, and won't cast otherwise:
```python
import pyarrow as pa
import pyarrow.parquet as pq

table = pa.csv.read_csv("test.csv")
table.cast(my_custom_schema)
pq.write_table(table, "test.parquet")
table_arrow_schema = pq.read_schema("test.parquet")
table_arrow_schema
```
gives
```
my_int: int64
animal: string
my_email: string
my_datetime: timestamp[ms]
my_date: date32[day]
```

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Research direction

Start with the Python repro using pa.csv.read_csv, table.cast, pq.write_table, and pq.read_schema, comparing the supplied my_custom_schema with the resulting Parquet schema. Trace the Parquet date conversion and verify that a date64[ms] column remains date64[ms] after the write/read roundtrip.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.