apache / apache/arrow

[C++][Dataset] Writing partitions with timestamp type give mis-formatted (integers) directory names

Open
#27,271 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

```Java

In [1]: df = pd.DataFrame({"dates": list(pd.date_range("2012-01-01", periods=2, freq="D")) * 5, "col": range(10)})

In [2]: table = pa.table(df)

In [8]: part = ds.partitioning(table.select(["dates"]).schema, flavor="hive")

In [9]: ds.write_dataset(table, "test_partition_timestamp", format="parquet", partitioning=part)

In [10]: !ls test_partition_timestamp/
'dates=1325376000000000000' 'dates=1325462400000000000'
```

Same happens for non-ns timestamp types.

(didn't directly find an existing issue about it)

**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-11378) / @jorisvandenbossche
#### Related issues:
- [[C++][Dataset] Support reading date/time-partitioned datasets accounting for URL encoding (Spark)](https://github.com/apache/arrow/issues/28395) (is related to)
- [[C++][Compute] Add strftime kernel](https://github.com/apache/arrow/issues/28874) (depends upon)

**Note**: *This issue was originally created as [ARROW-11378](https://issues.apache.org/jira/browse/ARROW-11378). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the ds.write_dataset reproduction and the Hive partitioning setup shown in the issue, then follow the related strftime-kernel issue because this work depends on it. Done means timestamp partitions are written with correctly formatted directory names for both nanosecond and non-nanosecond timestamp types, with behavior checked against the reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, pandas, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.