apache / apache/paimon

[Feature] pypaimon supports overwrite writes.

Open
#6,179 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

In offline model inference and table writing scenarios, it is common to conduct multiple offline evaluation experiments. It is necessary to support overwrite writes so that the results from the last run are retained. Currently, this is not supported in the code. For example:
```python
def overwrite(self, partition, commit_messages: List[CommitMessage], commit_identifier: int):
"""Commit the given commit messages in overwrite mode."""
raise RuntimeError("overwrite unsupported yet")
```

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating pypaimon's overwrite method shown in the issue and tracing the table-writing and commit paths around it. Determine how overwrite should retain results from the latest evaluation run, then identify the relevant tests or add coverage for the overwrite behavior; done means overwrite writes no longer raise the unsupported error and the expected previous results are replaced.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.