apache / apache/arrow-cookbook

[Python] Add recipe for appending/replacing data set partitions

Open
#278 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
108
Forks
49
Avg merge
2h 14m
Merged PRs (30d)
1

Description

The pyarrow [docs](https://arrow.apache.org/docs/python/generated/pyarrow.dataset.write_dataset.html#pyarrow.dataset.write_dataset) for the `exisiting_data_behavior` param contains this hint:

> This behavior, in combination with a unique basename_template for each write, will allow for an append workflow.
>
>‘delete_matching’ is useful when you are writing a partitioned dataset. The first time each partition directory is encountered the entire directory will be deleted. This allows you to overwrite old partitions completely.

A fully formed recipe for this would be nice as this hint is a bit hidden.

Contributor guide

Open the contributing guide

Research direction

The issue points to pyarrow.dataset.write_dataset and its existing_data_behavior parameter; start by reading that API documentation and the append and delete_matching hints. Add a cookbook recipe covering appending and replacing partitioned data, with examples that make the completed workflows clear.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.