Race condition on database upgrade
- Dominant language
- Elm
- Stars
- 2.3k
- Forks
- 184
- Avg merge
- 8h 2m
- Merged PRs (30d)
- 4
Description
When I upgrade Docspell on my Kubernetes deployment it looks like restserver and joex wants to upgrade the database on startup and lock each other to the point where I have to restart the database completely.
Output on both of the applications is the same:
```
[ioapp-compute-0] ERROR o.f.c.i.d.p.PostgreSQLAdvisoryLockTemplate - Unable to release PostgreSQL advisory lock
[ioapp-compute-0] INFO o.h.c.PoolManager - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
org.flywaydb.core.api.FlywayException: Number of retries exceeded while attempting to acquire PostgreSQL advisory lock. Configure the number of retries with the 'lockRetryCount' configuration option: https://flywaydb.org/documentation/configuration/parameters/lockRetryCount
at org.flywaydb.core.internal.strategy.RetryStrategy.doWithRetries(RetryStrategy.java:87)
at org.flywaydb.core.internal.database.postgresql.PostgreSQLAdvisoryLockTemplate.lock(PostgreSQLAdvisoryLockTemplate.java:86)
at org.flywaydb.core.internal.database.postgresql.PostgreSQLAdvisoryLockTemplate.execute(PostgreSQLAdvisoryLockTemplate.java:68)
at org.flywaydb.core.internal.database.postgresql.PostgreSQLConnection.lock(PostgreSQLConnection.java:99)
at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:141)
at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:191)
at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:152)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:216)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:165)
at org.flywaydb.core.Flyway.execute(Flyway.java:570)
at org.flywaydb.core.Flyway.migrate(Flyway.java:165)
at docspell.store.migrate.FlywayMigrate$.$anonfun$run$1(FlywayMigrate.scala:36)
at map @ docspell.store.impl.StoreImpl.migrate(StoreImpl.scala:25)
at liftF @ docspell.joex.JoexServer$.stream(JoexServer.scala:31)
at map @ fs2.internal.CompileScope.interruptibleEval(CompileScope.scala:413)
at flatMap @ fs2.internal.FreeC$.go$1(Algebra.scala:503)
at flatMap @ fs2.internal.FreeC$.$anonfun$compile$7(Algebra.scala:463)
at flatMap @ fs2.internal.FreeC$.go$1(Algebra.scala:460)
at flatMap @ fs2.internal.FreeC$.$anonfun$compile$7(Algebra.scala:463)
at flatMap @ fs2.internal.FreeC$.go$1(Algebra.scala:460)
at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:436)
at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:436)
at flatMap @ fs2.internal.CompileScope.$anonfun$acquireResource$4(CompileScope.scala:185)
at flatten @ fs2.internal.ScopedResource$$anon$1.acquired(ScopedResource.scala:139)
at flatMap @ fs2.internal.CompileScope.$anonfun$acquireResource$1(CompileScope.scala:183)
at flatMap @ fs2.internal.CompileScope.acquireResource(CompileScope.scala:180)
at flatMap @ fs2.internal.FreeC$.$anonfun$compile$12(Algebra.scala:511)
at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:436)
```
And on the database:
```
2021-03-14 08:51:16.866 UTC [9361] WARNING: you don't own a lock of type ExclusiveLock
```
After a restart of the database and both apps everything comes up in a clean way.
As a sidenote: The [deployment of the two apps is configured as `Recreate` strategy](https://git.tbrnt.ch/tobru/gitops-tbrnt/src/branch/master/docspell/restserver/deployment.yaml#L13) which means that the old version of the application is stopped before the new image is started again, that makes sure that the old version doesn't access the database the same time as the new version is starting up and upgrading the database.
Contributor guide
Assessment
This issue has not been assessed yet.