apache / apache/paimon

[Bug] Apache Paimon schema evolution

Open
#5,004 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

Not sure that it's a bag, but wanted to clarify the question of schema evolution.

Currently we are onboarding Paimon in way:
1. There is a Flink application which injects data to Paimon tables
2. There are readers which consumes the data later

During experimenting with schema evolution noticed that when schema change was done in external way Flink application didn't pick up changed properties even after it was restarted. For application to pick up the change I had to restart job without its state. What riese a question how to properly manage the schema evolution, should that be imbedded to the application as pre job execution steps or should that be managed somehow else?

### Compute Engine

Flink

### Minimal reproduce step

1. Write to Paimon table from stream application in way:

```
val table = env.sqlQuery(query)
val data = env.toChangelogStream(table)

val catalog = FlinkCatalogFactory.createPaimonCatalog(...)
val pTable = catalog.getTable(Identifier.create(..., ...))

new FlinkSinkBuilder(pTable)
.forRow(data, table.getSchema.toRowDataType)
.parallelism(1)
.build()
```
Usage of `HiveCatalog` didn't change the behaviour.

2. Change Paimon table property with external Java application based on Catalog API documentation section

### What doesn't meet your expectations?

Absence of understanding after consulting with documentation how safely evolve table schema. In our case it was attempt to change `snapshot.time-retained`. And preferably do it without losing job state

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the provided Flink streaming reproduction and the Catalog API documentation section used to change snapshot.time-retained. Compare behavior with and without restored job state, then document the supported schema-evolution procedure and whether the change can preserve state.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.