eikek / eikek/docspell

Race condition on database upgrade

Open
#716 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Elm
Stars
2.3k
Forks
184
Avg merge
8h 2m
Merged PRs (30d)
4

Description

When I upgrade Docspell on my Kubernetes deployment it looks like restserver and joex wants to upgrade the database on startup and lock each other to the point where I have to restart the database completely.

Output on both of the applications is the same:

```
[ioapp-compute-0] ERROR o.f.c.i.d.p.PostgreSQLAdvisoryLockTemplate - Unable to release PostgreSQL advisory lock
[ioapp-compute-0] INFO o.h.c.PoolManager - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
org.flywaydb.core.api.FlywayException: Number of retries exceeded while attempting to acquire PostgreSQL advisory lock. Configure the number of retries with the 'lockRetryCount' configuration option: https://flywaydb.org/documentation/configuration/parameters/lockRetryCount
at org.flywaydb.core.internal.strategy.RetryStrategy.doWithRetries(RetryStrategy.java:87)
at org.flywaydb.core.internal.database.postgresql.PostgreSQLAdvisoryLockTemplate.lock(PostgreSQLAdvisoryLockTemplate.java:86)
at org.flywaydb.core.internal.database.postgresql.PostgreSQLAdvisoryLockTemplate.execute(PostgreSQLAdvisoryLockTemplate.java:68)
at org.flywaydb.core.internal.database.postgresql.PostgreSQLConnection.lock(PostgreSQLConnection.java:99)
at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:141)
at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:191)
at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:152)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:216)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:165)
at org.flywaydb.core.Flyway.execute(Flyway.java:570)
at org.flywaydb.core.Flyway.migrate(Flyway.java:165)
at docspell.store.migrate.FlywayMigrate$.$anonfun$run$1(FlywayMigrate.scala:36)
at map @ docspell.store.impl.StoreImpl.migrate(StoreImpl.scala:25)
at liftF @ docspell.joex.JoexServer$.stream(JoexServer.scala:31)
at map @ fs2.internal.CompileScope.interruptibleEval(CompileScope.scala:413)
at flatMap @ fs2.internal.FreeC$.go$1(Algebra.scala:503)
at flatMap @ fs2.internal.FreeC$.$anonfun$compile$7(Algebra.scala:463)
at flatMap @ fs2.internal.FreeC$.go$1(Algebra.scala:460)
at flatMap @ fs2.internal.FreeC$.$anonfun$compile$7(Algebra.scala:463)
at flatMap @ fs2.internal.FreeC$.go$1(Algebra.scala:460)
at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:436)
at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:436)
at flatMap @ fs2.internal.CompileScope.$anonfun$acquireResource$4(CompileScope.scala:185)
at flatten @ fs2.internal.ScopedResource$$anon$1.acquired(ScopedResource.scala:139)
at flatMap @ fs2.internal.CompileScope.$anonfun$acquireResource$1(CompileScope.scala:183)
at flatMap @ fs2.internal.CompileScope.acquireResource(CompileScope.scala:180)
at flatMap @ fs2.internal.FreeC$.$anonfun$compile$12(Algebra.scala:511)
at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:436)
```

And on the database:

```
2021-03-14 08:51:16.866 UTC [9361] WARNING: you don't own a lock of type ExclusiveLock
```

After a restart of the database and both apps everything comes up in a clean way.

As a sidenote: The [deployment of the two apps is configured as `Recreate` strategy](https://git.tbrnt.ch/tobru/gitops-tbrnt/src/branch/master/docspell/restserver/deployment.yaml#L13) which means that the old version of the application is stopped before the new image is started again, that makes sure that the old version doesn't access the database the same time as the new version is starting up and upgrading the database.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.