apache / apache/paimon

[Bug] Saving dataframe in overwrite mode using Spark doesn't set primary key

Open
#7,687 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

3.4

### Compute Engine

Spark

### Minimal reproduce step

Saving dataframe with append mode is correctly setting the primary key in the schema:
Image

Schema:
Image

But when I try to save dataframe with overwrite mode then primary key is NOT set in the schema:
Image

Schema:
Image

### What doesn't meet your expectations?

Primary key in is NOT set in paimon table schema

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported Spark DataFrame save behavior on Paimon 3.4, comparing append mode with overwrite mode and inspecting the resulting table schema. Trace the Spark overwrite save path to find where primary-key metadata is lost; done means overwrite mode preserves the primary key in the Paimon table schema.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.