eikek / eikek/docspell

Database upgrade from 0.32.0 fails

Open
#1,660 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Elm
Stars
2.3k
Forks
184
Avg merge
8h 2m
Merged PRs (30d)
4

Description

Hei there,

I didn't update my instance for a while and sat on v0.32.0. I read all the release notes and the DB migration issue from 0.32.0 to 0.33.0 and understood that it was solved in 0.34.0, so I tried to go to 0.38.0 right away.

Sadly, the migration to the new file ids seems to fail on me for a different reason than mentioned before in #1436. I get the following error and I get it for an upgrade to 0.33.0, 0.34.0 or 0.38.0:

```text
Jul 23, 2022 12:08:37 AM org.jline.utils.Log logr
WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
2022.07.23 00:08:37:681 io-compute-4 INFO docspell.config.ConfigFactory.default:44
Using config from environment variables!
2022.07.23 00:08:38:0000 [io-comp...] [INFO ] docspell.restserver.Main.run:33 -
***> ______ _ _
***> | _ \ | | |
***> | | | |___ ___ ___ _ __ ___| | |
***> | | | / _ \ / __/ __| '_ \ / _ \ | |
***> | |/ / (_) | (__\__ \ |_) | __/ | |
***> |___/ \___/ \___|___/ .__/ \___|_|_|
***> | |
***> |_| v0.33.0 (#aafa73e7)
***> << REST Server >>
***> Id: rest1
***> Base-Url: http://localhost:7880
***> Database: jdbc:postgresql://db:5432/dbname
***> Fts: http://docspell-solr:8983/solr/docspell
***> Config:
***>
2022.07.23 00:08:39:0000 [io-comp...] [INFO ] docspell.store.migrate.FlywayMigrate - Running db migrations...
2022.07.23 00:08:39:0001 [io-comp...] [INFO ] docspell.store.migrate.FlywayMigrate - Using migration locations: List(classpath:db/migration/postgresql, classpath:db/migration/common)
2022.07.23 00:08:39:0002 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - Flyway Community Edition 8.5.2 by Redgate
2022.07.23 00:08:39:0003 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - See what's new here: https://flywaydb.org/documentation/learnmore/releaseNotes#8.5.2
2022.07.23 00:08:39:0004 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter -
2022.07.23 00:08:39:0005 [io-comp...] [INFO ] org.flywaydb.core.internal.database.base.BaseDatabaseType - Database: jdbc:postgresql://db:5432/dbname (PostgreSQL 13.4)
2022.07.23 00:08:39:0006 [io-comp...] [INFO ] org.flywaydb.core.internal.command.DbValidate - Successfully validated 57 migrations (execution time 00:00.065s)
2022.07.23 00:08:39:0007 [io-comp...] [INFO ] org.flywaydb.core.internal.command.DbMigrate - Current version of schema "public": 1.32.2
2022.07.23 00:08:39:0008 [io-comp...] [INFO ] org.flywaydb.core.internal.command.DbMigrate - Migrating schema "public" to version "1.33.0 - reorganize file ids"
2022.07.23 00:08:39:0009 [io-comp...] [INFO ] org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor - DB: table "file_migration_temp" does not exist, skipping
2022.07.23 00:08:44:0000 [io-comp...] [ERROR] org.flywaydb.core.internal.command.DbMigrate - Migration of schema "public" to version "1.33.0 - reorganize file ids" failed! Changes successfully rolled back.
org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException: Migration V1.33.0__reorganize_file_ids.sql failed
-------------------------------------------------
SQL State : 23505
Error Code : 0
Message : ERROR: duplicate key value violates unique constraint "file_migration_temp_original_file_key"
Detail: Key (original_file)=(4w5aJANWpxNrVVkAJRwuFLyr37FqPy5fLyrEzX7hNtgG) already exists.
Location : db/migration/postgresql/V1.33.0__reorganize_file_ids.sql (/opt/file:/opt/docspell-restserver-0.33.0/lib/com.github.eikek.docspell-store-0.33.0.jar!/db/migration/postgresql/V1.33.0__reorganize_file_ids.sql)
Line : 11
Statement : -- Source files
insert into file_migration_temp (original_file, cid, category, new_file)
select
rs.file_id as original_file,
i.cid,
'attachmentsource' as category,
i.cid || '/attachmentsource/' || rs.file_id as new_file
from attachment_source rs
inner join attachment ra on rs.id = ra.attachid
inner join item i on ra.itemid = i.itemid

at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:385)
at org.flywaydb.core.internal.command.DbMigrate.lambda$applyMigrations$1(DbMigrate.java:275)
at org.flywaydb.core.internal.jdbc.TransactionalExecutionTemplate.execute(TransactionalExecutionTemplate.java:55)
at org.flywaydb.core.internal.command.DbMigrate.applyMigrations(DbMigrate.java:274)
at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:247)
at org.flywaydb.core.internal.command.DbMigrate.lambda$migrateAll$0(DbMigrate.java:141)
at org.flywaydb.core.internal.database.postgresql.PostgreSQLAdvisoryLockTemplate.execute(PostgreSQLAdvisoryLockTemplate.java:69)
at org.flywaydb.core.internal.database.postgresql.PostgreSQLConnection.lock(PostgreSQLConnection.java:99)
at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:139)
at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:141)
at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:98)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:172)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:124)
at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:205)
at org.flywaydb.core.Flyway.migrate(Flyway.java:124)
at docspell.store.migrate.FlywayMigrate$.$anonfun$run$1(FlywayMigrate.scala:41)
at delay @ docspell.store.migrate.FlywayMigrate$.run(FlywayMigrate.scala:21)
at map @ docspell.store.impl.StoreImpl.migrate(StoreImpl.scala:31)
Caused by: org.flywaydb.core.internal.sqlscript.FlywaySqlScriptException: Migration V1.33.0__reorganize_file_ids.sql failed
-------------------------------------------------
SQL State : 23505
Error Code : 0
Message : ERROR: duplicate key value violates unique constraint "file_migration_temp_original_file_key"
Detail: Key (original_file)=(4w5aJANWpxNrVVkAJRwuFLyr37FqPy5fLyrEzX7hNtgG) already exists.
Location : db/migration/postgresql/V1.33.0__reorganize_file_ids.sql (/opt/file:/opt/docspell-restserver-0.33.0/lib/com.github.eikek.docspell-store-0.33.0.jar!/db/migration/postgresql/V1.33.0__reorganize_file_ids.sql)
Line : 11
Statement : -- Source files
insert into file_migration_temp (original_file, cid, category, new_file)
select
rs.file_id as original_file,
i.cid,
'attachmentsource' as category,
i.cid || '/attachmentsource/' || rs.file_id as new_file
from attachment_source rs
inner join attachment ra on rs.id = ra.attachid
inner join item i on ra.itemid = i.itemid

at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.handleException(DefaultSqlScriptExecutor.java:275)
at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.executeStatement(DefaultSqlScriptExecutor.java:222)
at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.execute(DefaultSqlScriptExecutor.java:126)
at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.executeOnce(SqlMigrationExecutor.java:69)
at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.lambda$execute$0(SqlMigrationExecutor.java:58)
at org.flywaydb.core.internal.database.DefaultExecutionStrategy.execute(DefaultExecutionStrategy.java:27)
at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.execute(SqlMigrationExecutor.java:57)
at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:377)
at org.flywaydb.core.internal.command.DbMigrate.lambda$applyMigrations$1(DbMigrate.java:275)
at org.flywaydb.core.internal.jdbc.TransactionalExecutionTemplate.execute(TransactionalExecutionTemplate.java:55)
at org.flywaydb.core.internal.command.DbMigrate.applyMigrations(DbMigrate.java:274)
at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:247)
at org.flywaydb.core.internal.command.DbMigrate.lambda$migrateAll$0(DbMigrate.java:141)
at org.flywaydb.core.internal.database.postgresql.PostgreSQLAdvisoryLockTemplate.execute(PostgreSQLAdvisoryLockTemplate.java:69)
at org.flywaydb.core.internal.database.postgresql.PostgreSQLConnection.lock(PostgreSQLConnection.java:99)
at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:139)
at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:141)
at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:98)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:172)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:124)
at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:205)
at org.flywaydb.core.Flyway.migrate(Flyway.java:124)
at docspell.store.migrate.FlywayMigrate$.$anonfun$run$1(FlywayMigrate.scala:41)
at cats.effect.IOFiber.runLoop(IOFiber.scala:358)
at cats.effect.IOFiber.execR(IOFiber.scala:1327)
at cats.effect.IOFiber.run(IOFiber.scala:139)
at cats.effect.unsafe.WorkerThread.run(WorkerThread.scala:461)
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "file_migration_temp_original_file_key"
Detail: Key (original_file)=(4w5aJANWpxNrVVkAJRwuFLyr37FqPy5fLyrEzX7hNtgG) already exists.
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:329)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:315)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:291)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:286)
at org.flywaydb.core.internal.jdbc.JdbcTemplate.executeStatement(JdbcTemplate.java:201)
at org.flywaydb.core.internal.sqlscript.ParsedSqlStatement.execute(ParsedSqlStatement.java:95)
at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.executeStatement(DefaultSqlScriptExecutor.java:210)

... 25 more
```
The container then restarts and the same error occurs again.

The database seems to be still intact, since it still works with 0.32.0. Which is good news.

Some more info:
- I deploy Docspell on Docker
- I have a database dump from before any of the tried upgrades.

I hope this is enough information to get started and thanks for looking into this in advance!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.