Provide way of DB schema migration between releases
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
### Describe the bug
Upgrading Polaris from 1.0.1 to 1.1.0 is virtually impossible without manually check difference between DB schemas.
Admin tool `boostrap` command is failing with:
```
java.lang.RuntimeException: Error executing sql script: Failed due to 'ERROR: column "location_without_scheme" does not exist
Position: 119' (error code 0, sql-state '42703'), after 1 attempts and 5000 milliseconds
at org.apache.polaris.persistence.relational.jdbc.JdbcMetaStoreManagerFactory.bootstrapRealms(JdbcMetaStoreManagerFactory.java:158)
at org.apache.polaris.persistence.relational.jdbc.JdbcMetaStoreManagerFactory_ClientProxy.bootstrapRealms(Unknown Source)
at org.apache.polaris.admintool.BootstrapCommand.call(BootstrapCommand.java:149)
at org.apache.polaris.admintool.BootstrapCommand.call(BootstrapCommand.java:32)
at picocli.CommandLine.executeUserObject(CommandLine.java:2031)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2469)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2461)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2423)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2425)
at io.quarkus.picocli.runtime.PicocliRunner$EventExecutionStrategy.execute(PicocliRunner.java:26)
at picocli.CommandLine.execute(CommandLine.java:2174)
at io.quarkus.picocli.runtime.PicocliRunner.run(PicocliRunner.java:40)
at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:141)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:80)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:51)
at io.quarkus.runner.GeneratedMain.main(Unknown Source)
at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:69)
at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:37)
Caused by: java.sql.SQLException: Failed due to 'ERROR: column "location_without_scheme" does not exist
Position: 119' (error code 0, sql-state '42703'), after 1 attempts and 5000 milliseconds
at org.apache.polaris.persistence.relational.jdbc.DatasourceOperations.withRetries(DatasourceOperations.java:300)
at org.apache.polaris.persistence.relational.jdbc.DatasourceOperations.runWithinTransaction(DatasourceOperations.java:206)
at org.apache.polaris.persistence.relational.jdbc.DatasourceOperations.executeScript(DatasourceOperations.java:90)
at org.apache.polaris.persistence.relational.jdbc.JdbcMetaStoreManagerFactory.bootstrapRealms(JdbcMetaStoreManagerFactory.java:152)
... 19 more
Caused by: java.lang.RuntimeException: org.postgresql.util.PSQLException: ERROR: column "location_without_scheme" does not exist
Position: 119
at org.apache.polaris.persistence.relational.jdbc.DatasourceOperations.lambda$executeScript$0(DatasourceOperations.java:106)
at org.apache.polaris.persistence.relational.jdbc.DatasourceOperations.lambda$runWithinTransaction$4(DatasourceOperations.java:214)
at org.apache.polaris.persistence.relational.jdbc.DatasourceOperations.withRetries(DatasourceOperations.java:271)
... 22 more
Caused by: org.postgresql.util.PSQLException: ERROR: column "location_without_scheme" does not exist
Position: 119
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2734)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2421)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:372)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:518)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:435)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:357)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:342)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:318)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:313)
at io.agroal.pool.wrapper.StatementWrapper.execute(StatementWrapper.java:220)
at org.apache.polaris.persistence.relational.jdbc.DatasourceOperations.lambda$executeScript$0(DatasourceOperations.java:104)
... 24 more
Bootstrap encountered errors during operation.
```
Digging into code revealed added `location_without_scheme` fields in schema v2. But polaris-admin doesn't provide schema migration feature.
### To Reproduce
_No response_
### Actual Behavior
_No response_
### Expected Behavior
_No response_
### Additional context
_No response_
### System information
_No response_
Contributor guide
Research direction
Start with the polaris-admin BootstrapCommand and JdbcMetaStoreManagerFactory.bootstrapRealms, then inspect how schema v2 added the location_without_scheme fields. Reproduce the PostgreSQL failure when bootstrapping a database from Polaris 1.0.1, and trace the schema scripts used by the bootstrap path. Done means an upgrade between releases can migrate the existing schema before bootstrap succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, postgresql
- Domain
- cli, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100