JetBrains / JetBrains/Exposed

Add retry mechanism alongside withDataBaseLock{}

Open
#796 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Kotlin
Stars
9.3k
Forks
798
Avg merge
4d 2h
Merged PRs (30d)
26

Description

Here's the error scenario:

* Multiple microservices connect to same SQL server
* During deployment, they are all spun up at nearly the same time
* Given `withDataBaseLock{}` blocks execution if there is already another lock on the database, it is possible that some microservice will not be able to execute the `withDataBaseLock{}` code.
* This becomes more and more likely, the more microservice instances there are

So, for example, it's possible that one poor microservice will not be able to create/update it's tables:

```
transaction {
withDataBaseLock{
SchemaUtils.createMissingTablesAndColumns(
Transactions,
Payments
)
}
}
```

As a result, when some API call comes and tries to find some table, it won't be able to do that. So it is a critical failure, but even worse, it fails silently.

A possible solution is throwing a runtimeException (though that's a bit harsh) so that microservices will keep dying and restarting until all other microservices are done with their database lock. So, eventually by forcing restarts, everyone will get to execute their create table calls etc. Though this is not a very elegant solution.

Another potential solution is to add a retry mechanism within withDatabaseLock. Something like:

```
transaction {
withDataBaseLock(retry = true, retryDelay = 30){
SchemaUtils.createMissingTablesAndColumns(
Transactions,
Payments
)
}
}
```

That is, if there is a lock, then wait 30 seconds and try again. By putting this inside a `runBlocking {}` coroutine block, I can be certain that the code inside all `withDataBaseLock{}` blocks across all microservices will eventually run within a few minutes and all microservices live happily ever after.

Does this seem possible @Tapac? (Or is there some other solution I'm not aware of?)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.