[C++][Dataset] Writing partitions with timestamp type give mis-formatted (integers) directory names
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
```Java
In [1]: df = pd.DataFrame({"dates": list(pd.date_range("2012-01-01", periods=2, freq="D")) * 5, "col": range(10)})
In [2]: table = pa.table(df)
In [8]: part = ds.partitioning(table.select(["dates"]).schema, flavor="hive")
In [9]: ds.write_dataset(table, "test_partition_timestamp", format="parquet", partitioning=part)
In [10]: !ls test_partition_timestamp/
'dates=1325376000000000000' 'dates=1325462400000000000'
```
Same happens for non-ns timestamp types.
(didn't directly find an existing issue about it)
**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-11378) / @jorisvandenbossche
#### Related issues:
- [[C++][Dataset] Support reading date/time-partitioned datasets accounting for URL encoding (Spark)](https://github.com/apache/arrow/issues/28395) (is related to)
- [[C++][Compute] Add strftime kernel](https://github.com/apache/arrow/issues/28874) (depends upon)
**Note**: *This issue was originally created as [ARROW-11378](https://issues.apache.org/jira/browse/ARROW-11378). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with the ds.write_dataset reproduction and the Hive partitioning setup shown in the issue, then follow the related strftime-kernel issue because this work depends on it. Done means timestamp partitions are written with correctly formatted directory names for both nanosecond and non-nanosecond timestamp types, with behavior checked against the reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100