apache / apache/paimon

[Feature] Add the flink jobid field to the files metadata table

Open
#4,490 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

When a similar error occurs, use jobid to determine whether multiple tasks are writing to the same table at the same time. Easy to find problems.
```
Trying to delete file data-b2fb3e14-65f8-4058-8d64-5e6a6834eb5d-0.avro which is not previously added
```

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the files metadata table and the code path that emits the "Trying to delete file" error. Determine how a Flink job ID could be associated with that metadata, then verify that the job ID is available for diagnosing concurrent writers; the issue does not name specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.