apache / apache/arrow

[Python] write_dataset how to add and update data

Open
#14,834 5 comments 1 reaction 0 assignees View on GitHub
Component: Parquet Component: Python Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the usage question you have. Please include as many useful details as possible.

First, save the parquet file, there are 5 pieces of data
```
dataset_name = 'test_update'
df = pd.DataFrame({'one': [-1, 3, 2.5, 2.5, 2.5],
'two': ['foo', 'bar', 'baz','foo','foo'],
'three': [True, False, True,False,False]},
)
table = pa.Table.from_pandas(df)
ds.write_dataset(table, dataset_name,
existing_data_behavior='overwrite_or_ignore',
format="parquet")
```

Then I want to add two new ones, and I want to get a total of 7 results, and the new data is as follows:
```
df = pd.DataFrame({'one': [1, 2],
'two': ['foo-insert1','foo-insert2'],
'three': [True, False]},
)

table = pa.Table.from_pandas(df)
ds.write_dataset(table, dataset_name,
# existing_data_behavior='delete_matching',
existing_data_behavior='overwrite_or_ignore',
format="parquet")
```
1. **But this overwrites the original, there are only two data, how to achieve new data on the basis of the original**

2. **I have another question, if I want to update the data according to the conditions, how to change how to do it, for example**

> Update one=-1, two=foo's three to False

- python=3.10
- pyarrow=10.0.0

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Research direction

Start with the ds.write_dataset calls in the issue and reproduce them using Python 3.10, PyArrow 10.0.0, and Parquet. Read the write_dataset documentation to determine append and conditional-update behavior; done means providing an authoritative explanation or documentation example for both requested operations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.