apache / apache/arrow

arrow dataset: how to use date.year and date.month as partitioning

Open
#14,619 10 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

The data columns:
datetime, value

I want to build a dataset with partitioning like ("date.year", "date.month"). How to do that?
And when do filtering like date > "2021-09-02" and date < "2022-04-06", will it read the partition files between 2021-09 to 2022-04 only?

Thanks

Contributor guide

Open the contributing guide

Research direction

No file, test, or entry point is named in the issue. Start by locating the Apache Arrow dataset partitioning documentation and determine how date.year/date.month partitioning and date-range filtering are supported; done means documenting a concrete usage pattern and the expected partition pruning behavior.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.