Add retry mechanism alongside withDataBaseLock{}
- Dominant language
- Kotlin
- Stars
- 9.3k
- Forks
- 798
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 26
Description
Here's the error scenario:
* Multiple microservices connect to same SQL server
* During deployment, they are all spun up at nearly the same time
* Given `withDataBaseLock{}` blocks execution if there is already another lock on the database, it is possible that some microservice will not be able to execute the `withDataBaseLock{}` code.
* This becomes more and more likely, the more microservice instances there are
So, for example, it's possible that one poor microservice will not be able to create/update it's tables:
```
transaction {
withDataBaseLock{
SchemaUtils.createMissingTablesAndColumns(
Transactions,
Payments
)
}
}
```
As a result, when some API call comes and tries to find some table, it won't be able to do that. So it is a critical failure, but even worse, it fails silently.
A possible solution is throwing a runtimeException (though that's a bit harsh) so that microservices will keep dying and restarting until all other microservices are done with their database lock. So, eventually by forcing restarts, everyone will get to execute their create table calls etc. Though this is not a very elegant solution.
Another potential solution is to add a retry mechanism within withDatabaseLock. Something like:
```
transaction {
withDataBaseLock(retry = true, retryDelay = 30){
SchemaUtils.createMissingTablesAndColumns(
Transactions,
Payments
)
}
}
```
That is, if there is a lock, then wait 30 seconds and try again. By putting this inside a `runBlocking {}` coroutine block, I can be certain that the code inside all `withDataBaseLock{}` blocks across all microservices will eventually run within a few minutes and all microservices live happily ever after.
Does this seem possible @Tapac? (Or is there some other solution I'm not aware of?)
Contributor guide
Assessment
This issue has not been assessed yet.