[Bug] Saving dataframe in overwrite mode using Spark doesn't set primary key
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
3.4
### Compute Engine
Spark
### Minimal reproduce step
Saving dataframe with append mode is correctly setting the primary key in the schema:
Schema:
But when I try to save dataframe with overwrite mode then primary key is NOT set in the schema:
Schema:
### What doesn't meet your expectations?
Primary key in is NOT set in paimon table schema
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reported Spark DataFrame save behavior on Paimon 3.4, comparing append mode with overwrite mode and inspecting the resulting table schema. Trace the Spark overwrite save path to find where primary-key metadata is lost; done means overwrite mode preserves the primary key in the Paimon table schema.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100