debezium / debezium/dbz

Graceful shutdown not working when Postgres LSNs need flushing [DBZ-3351]

Open
#445 0 comments 0 reactions 0 assignees View on GitHub
component/embedded-engine migrated-from-jira type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-3351](https://issues.redhat.com/browse/DBZ-3351)

There is an ordering issue with graceful shutdown on Postgres databases using Embedded engine. The replication stream is closed before flush.

The following things happen when EmbeddedEngine.close() is called:
# The PostgresStreamingChangeEventSource.execute(..) method is terminated and the following finally block is triggered.
See LOC https://github.com/debezium/debezium/blob/master/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/PostgresStreamingChangeEventSource.java#L152-L173
{code:java}
public void execute(ChangeEventSourceContext context) throws InterruptedException {
...
finally {
if (replicationConnection != null) {
...
replicationStream.set(null);
}
}
}
{code}
# In EmbeddedEngine the finally block of the run method is invoked and this code block is executed:
See LOC:  https://github.com/debezium/debezium/blob/86cb71ad8b063a66fff8148a9a2dc94830d21d09/debezium-embedded/src/main/java/io/debezium/embedded/EmbeddedEngine.java#L838-L850
{code:java}
finally {
...
// Always commit offsets that were captured from the source records we actually processed ...
commitOffsets(offsetWriter, commitTimeout, task);
...
}
{code}
# This triggers the execution of the PostgresStreamingChangeEventSource commitOffset method, but replicationStream was set to null in step 1, so the flushLsn never happens:
See LOC: https://github.com/debezium/debezium/blob/master/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/PostgresStreamingChangeEventSource.java#L334-L347
{code:java}
@Override
public void commitOffset(Map offset) {
try {
...
if (replicationStream != null && lsn != null) {
...
// tell the server the point up to which we've processed data, so it can be free to recycle WAL segments
replicationStream.flushLsn(lsn);
}
...
}
{code}
# The shutdown sequence completes with no exceptions raised, despite the complete failure to flush LSNs

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.