Graceful shutdown not working when Postgres LSNs need flushing [DBZ-3351]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-3351](https://issues.redhat.com/browse/DBZ-3351)
There is an ordering issue with graceful shutdown on Postgres databases using Embedded engine. The replication stream is closed before flush.
The following things happen when EmbeddedEngine.close() is called:
# The PostgresStreamingChangeEventSource.execute(..) method is terminated and the following finally block is triggered.
See LOC https://github.com/debezium/debezium/blob/master/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/PostgresStreamingChangeEventSource.java#L152-L173
{code:java}
public void execute(ChangeEventSourceContext context) throws InterruptedException {
...
finally {
if (replicationConnection != null) {
...
replicationStream.set(null);
}
}
}
{code}
# In EmbeddedEngine the finally block of the run method is invoked and this code block is executed:
See LOC: https://github.com/debezium/debezium/blob/86cb71ad8b063a66fff8148a9a2dc94830d21d09/debezium-embedded/src/main/java/io/debezium/embedded/EmbeddedEngine.java#L838-L850
{code:java}
finally {
...
// Always commit offsets that were captured from the source records we actually processed ...
commitOffsets(offsetWriter, commitTimeout, task);
...
}
{code}
# This triggers the execution of the PostgresStreamingChangeEventSource commitOffset method, but replicationStream was set to null in step 1, so the flushLsn never happens:
See LOC: https://github.com/debezium/debezium/blob/master/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/PostgresStreamingChangeEventSource.java#L334-L347
{code:java}
@Override
public void commitOffset(Map offset) {
try {
...
if (replicationStream != null && lsn != null) {
...
// tell the server the point up to which we've processed data, so it can be free to recycle WAL segments
replicationStream.flushLsn(lsn);
}
...
}
{code}
# The shutdown sequence completes with no exceptions raised, despite the complete failure to flush LSNs
Contributor guide
Assessment
This issue has not been assessed yet.