[Python][C++] pyarrow parquet writer - missing time logical type
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
pyarrow.parquet.write_table used for write Parquet file
In parquet schema missed logical type for TIME, it’s just long type
PyArrow Schema
```java
NUMBER: int64
DECIMAL: int64
NUMERIC: int64
INT: int64
FLOAT: double
VARCHAR: string
TEXT: string
CHAR: string
BOOLEAN: bool
ARR: string
VAR: string
OBJ: string
TIMESTAMP: timestamp[ns]
DATE: date64[ms]
TIME: time64[ns]
PK: int64
UUID: binary
UUID2: string
UUID3: string
```
Parquet schema
```java
{
"type" : "record",
"name" : "schema",
"fields" : [ {
"name" : "NUMBER",
"type" : [ "null", "long" ],
"default" : null
}, {
"name" : "DECIMAL",
"type" : [ "null", "long" ],
"default" : null
}, {
"name" : "NUMERIC",
"type" : [ "null", "long" ],
"default" : null
}, {
"name" : "INT",
"type" : [ "null", "long" ],
"default" : null
}, {
"name" : "FLOAT",
"type" : [ "null", "double" ],
"default" : null
}, {
"name" : "VARCHAR",
"type" : [ "null", "string" ],
"default" : null
}, {
"name" : "TEXT",
"type" : [ "null", "string" ],
"default" : null
}, {
"name" : "CHAR",
"type" : [ "null", "string" ],
"default" : null
}, {
"name" : "BOOLEAN",
"type" : [ "null", "boolean" ],
"default" : null
}, {
"name" : "ARR",
"type" : [ "null", "string" ],
"default" : null
}, {
"name" : "VAR",
"type" : [ "null", "string" ],
"default" : null
}, {
"name" : "OBJ",
"type" : [ "null", "string" ],
"default" : null
}, {
"name" : "TIMESTAMP",
"type" : [ "null", {
"type" : "long",
"logicalType" : "timestamp-micros"
} ],
"default" : null
}, {
"name" : "DATE",
"type" : [ "null", {
"type" : "int",
"logicalType" : "date"
} ],
"default" : null
}, {
"name" : "TIME",
"type" : [ "null", "long" ],
"default" : null
}, {
"name" : "PK",
"type" : [ "null", "long" ],
"default" : null
}, {
"name" : "UUID",
"type" : [ "null", "bytes" ],
"default" : null
}, {
"name" : "UUID2",
"type" : [ "null", "string" ],
"default" : null
}, {
"name" : "UUID3",
"type" : [ "null", "string" ],
"default" : null
} ]
}
```
**Reporter**: [Vadim Goy](https://issues.apache.org/jira/browse/ARROW-17506)
**Assignee**: [Quang Hoang](https://issues.apache.org/jira/browse/ARROW-17506) / @quanghgx
**Note**: *This issue was originally created as [ARROW-17506](https://issues.apache.org/jira/browse/ARROW-17506). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the issue through pyarrow.parquet.write_table with a column corresponding to the reported time64[ns] type, then inspect the generated Parquet schema alongside the PyArrow schema shown here. Done means the TIME field retains an appropriate Parquet logical type rather than appearing only as long; the payload names no source file or test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100