PostgreSQL connector task got stuck in "RUNNING" state for 30 minutes after unhandled exception [DBZ-944]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-944](https://issues.redhat.com/browse/DBZ-944)
If an exception was thrown during task execution, then Task stucks in running state. After 30 minutes it changes its state to failed state. During that period we cannot detect failure and restart the task.
*Expected result:*
Task changes its state to failed just after exception been thrown.
*Actual result:*
Task changes its state to failed in 30 minutes after exception been thrown.
*Some investigation on this issue:*
1. PostgresConnectorTask.commit() is called from kafka connect code just after the exception was thrown.
2. PostgresConnectorTask.commit() is blocked in RecordsStreamProducer.commit() call.
3. RecordsStreamProducer.commit() is awaiting for lock from RecordsStreamProducer.streamChanges(): the actual lock is in org.postgresql.core.v3.CopyDualImpl writeToCopy vs readFromCopy
4. Connect thread stacks just after exception are in "blocked.txt" attachment
5. Connect thread stacks 30 min after exception are in "unblocked.txt" attachment
6. Connect logs are in "connect_log.txt" attachment(exception was thrown in 12:50, and task failed only in 13:20)
7. The problem is reproduced in 100% test runs(ses Steps to Reproduce)
Need help guys!
Contributor guide
Assessment
This issue has not been assessed yet.