debezium / debezium/dbz

PostgreSQL connector task got stuck in "RUNNING" state for 30 minutes after unhandled exception [DBZ-944]

Open
#175 0 comments 0 reactions 0 assignees View on GitHub
component/postgresql-connector migrated-from-jira type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-944](https://issues.redhat.com/browse/DBZ-944)

If an exception was thrown during task execution, then Task stucks in running state. After 30 minutes it changes its state to failed state. During that period we cannot detect failure and restart the task.

*Expected result:*
Task changes its state to failed just after exception been thrown.

*Actual result:*
Task changes its state to failed in 30 minutes after exception been thrown.

*Some investigation on this issue:*
1. PostgresConnectorTask.commit() is called from kafka connect code just after the exception was thrown.
2. PostgresConnectorTask.commit() is blocked in RecordsStreamProducer.commit() call.
3. RecordsStreamProducer.commit() is awaiting for lock from RecordsStreamProducer.streamChanges(): the actual lock is in org.postgresql.core.v3.CopyDualImpl writeToCopy vs readFromCopy
4. Connect thread stacks just after exception are in "blocked.txt" attachment
5. Connect thread stacks 30 min after exception are in "unblocked.txt" attachment
6. Connect logs are in "connect_log.txt" attachment(exception was thrown in 12:50, and task failed only in 13:20)
7. The problem is reproduced in 100% test runs(ses Steps to Reproduce)

Need help guys!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.