apache / apache/arrow

[Python][C++] pyarrow parquet writer - missing time logical type

Open
#32,766 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

 

pyarrow.parquet.write_table used for write Parquet file
In parquet schema missed logical type for TIME, it’s just long type

PyArrow Schema
```java

NUMBER: int64
DECIMAL: int64
NUMERIC: int64
INT: int64
FLOAT: double
VARCHAR: string
TEXT: string
CHAR: string
BOOLEAN: bool
ARR: string
VAR: string
OBJ: string
TIMESTAMP: timestamp[ns]
DATE: date64[ms]
TIME: time64[ns]
PK: int64
UUID: binary
UUID2: string
UUID3: string
```

Parquet schema
```java

{
  "type" : "record",
  "name" : "schema",
  "fields" : [ {
    "name" : "NUMBER",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "DECIMAL",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "NUMERIC",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "INT",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "FLOAT",
    "type" : [ "null", "double" ],
    "default" : null
  }, {
    "name" : "VARCHAR",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "TEXT",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "CHAR",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "BOOLEAN",
    "type" : [ "null", "boolean" ],
    "default" : null
  }, {
    "name" : "ARR",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "VAR",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "OBJ",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "TIMESTAMP",
    "type" : [ "null", {
      "type" : "long",
      "logicalType" : "timestamp-micros"
    } ],
    "default" : null
  }, {
    "name" : "DATE",
    "type" : [ "null", {
      "type" : "int",
      "logicalType" : "date"
    } ],
    "default" : null
  }, {
    "name" : "TIME",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "PK",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "UUID",
    "type" : [ "null", "bytes" ],
    "default" : null
  }, {
    "name" : "UUID2",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "UUID3",
    "type" : [ "null", "string" ],
    "default" : null
  } ]
}
```
 

 

 

**Reporter**: [Vadim Goy](https://issues.apache.org/jira/browse/ARROW-17506)
**Assignee**: [Quang Hoang](https://issues.apache.org/jira/browse/ARROW-17506) / @quanghgx

**Note**: *This issue was originally created as [ARROW-17506](https://issues.apache.org/jira/browse/ARROW-17506). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue through pyarrow.parquet.write_table with a column corresponding to the reported time64[ns] type, then inspect the generated Parquet schema alongside the PyArrow schema shown here. Done means the TIME field retains an appropriate Parquet logical type rather than appearing only as long; the payload names no source file or test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.