apache / apache/arrow-cookbook
[Python] Add recipe for appending/replacing data set partitions
- Dominant language
- C++
- Stars
- 108
- Forks
- 49
- Avg merge
- 2h 14m
- Merged PRs (30d)
- 1
Description
The pyarrow [docs](https://arrow.apache.org/docs/python/generated/pyarrow.dataset.write_dataset.html#pyarrow.dataset.write_dataset) for the `exisiting_data_behavior` param contains this hint:
> This behavior, in combination with a unique basename_template for each write, will allow for an append workflow.
>
>‘delete_matching’ is useful when you are writing a partitioned dataset. The first time each partition directory is encountered the entire directory will be deleted. This allows you to overwrite old partitions completely.
A fully formed recipe for this would be nice as this hint is a bit hidden.
Contributor guide
Research direction
The issue points to pyarrow.dataset.write_dataset and its existing_data_behavior parameter; start by reading that API documentation and the append and delete_matching hints. Add a cookbook recipe covering appending and replacing partitioned data, with examples that make the completed workflows clear.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100