[C++] date64[ms] comes back as date32[day] after roundtrip to Parquet
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I'm trying to build a tool to confirm that a given parquet file conforms to an expected schema with [our metadata format](https://github.com/moj-analytical-services/mojap-metadata). There appears to be a bug with how pyarrow converts dates, however.
I have a dummy dataset, called test.csv (expand to view)
|my_int|animal |my_email |my_datetime |my_date |
|------|-------|-------------------------|-------------------|----------|
|16 |NA |omar65@waller.info |2013-01-14 15:54:20|1993-09-27|
|13 |cat |nataliechavez@hotmail.com|2006-12-16 01:44:21|1989-04-08|
|13 |dog |stephanie12@leblanc.com |1972-06-19 11:22:59|2006-10-01|
|18 |NA |cameronscott@bradshaw.com|2009-07-11 00:45:28|1992-07-24|
|10 |fish |rossrebecca@gould.net |1970-09-11 22:26:13|2011-11-30|
|13 |cat |escobarryan@gmail.com |2018-09-18 16:18:37|1979-01-22|
|16 |dog |shelly51@mitchell.com |2002-08-26 10:25:03|1981-03-27|
|11 |dog |kwilliams@hotmail.com |1989-02-26 16:33:37|2012-06-19|
|19 |chicken|lisacharles@kennedy.com |1974-07-25 04:41:22|2003-11-06|
|11 |chicken|olsonchelsea@gray.org |1992-04-03 09:47:30|1972-07-06|
I have a user-generated schema that resolves to this:
```
my_int: int64
animal: string
my_email: string
my_datetime: timestamp[s]
my_date: date64[ms]
```
However, the last column seems to be read by pyarrow as `date32[day]`, and won't cast otherwise:
```python
import pyarrow as pa
import pyarrow.parquet as pq
table = pa.csv.read_csv("test.csv")
table.cast(my_custom_schema)
pq.write_table(table, "test.parquet")
table_arrow_schema = pq.read_schema("test.parquet")
table_arrow_schema
```
gives
```
my_int: int64
animal: string
my_email: string
my_datetime: timestamp[ms]
my_date: date32[day]
```
### Component(s)
Parquet, Python
Contributor guide
Research direction
Start with the Python repro using pa.csv.read_csv, table.cast, pq.write_table, and pq.read_schema, comparing the supplied my_custom_schema with the resulting Parquet schema. Trace the Parquet date conversion and verify that a date64[ms] column remains date64[ms] after the write/read roundtrip.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100