arrow dataset: how to use date.year and date.month as partitioning
Open
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
The data columns:
datetime, value
I want to build a dataset with partitioning like ("date.year", "date.month"). How to do that?
And when do filtering like date > "2021-09-02" and date < "2022-04-06", will it read the partition files between 2021-09 to 2022-04 only?
Thanks
Contributor guide
Research direction
No file, test, or entry point is named in the issue. Start by locating the Apache Arrow dataset partitioning documentation and determine how date.year/date.month partitioning and date-range filtering are supported; done means documenting a concrete usage pattern and the expected partition pruning behavior.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100