debezium / debezium/dbz

IBMi (db2as400): StackOverflowError from close()/connection()/cancelJob() mutual recursion when host disconnects mid journal retrieval

Open
#2,600 1 comment 0 reactions 0 assignees View on GitHub
component/ibmi-connector type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

## Bug report

**What Debezium connector do you use and what version?**

`The IBMi connector, io.debezium.connector.db2as400.As400RpcConnector, on Debezium 3.4.0.Final.`

---

**What is the connector configuration?**

`Journal-based CDC against IBM i / DB2 for i, one task per connector, with error tolerance and infinite retries configured. Non-secret settings relevant to the failure:

connector.class = io.debezium.connector.db2as400.As400RpcConnector
tasks.max = 1
errors.tolerance = all
errors.retry.timeout = -1
max.retries = -1
retriable.restart.connector.wait.ms = 10000
database.connection.timeout.ms = 900000
socket.timeout = 900000

Note: retries/error tolerance are configured but do not prevent the failure, because the throwable is a java.lang.Error (see below).`

---

**What is the captured database version and mode of deployment?**

_E.g. on-premises, with a specific cloud provider, etc._

`IBM i (DB2 for i), journal-based CDC (QjoRetrieveJournalEntries). On-premises IBM i hosts; Debezium runs in Kafka Connect (distributed mode). The failure is reliably triggered when the IBM i host goes offline for a nightly database backup while streaming is active.`

---

**What behavior do you expect?**

`When the IBM i host becomes unreachable during an in-flight journal retrieval, the connector should detect the lost connection, tear down the RPC connection cleanly (without opening a new connection solely to cancel a job that can no longer be cancelled), and surface a retriable error so the connector retries per its configured backoff and resumes when the host returns.`

---

**What behavior do you see?**

`The connector enters unbounded mutual recursion on its own teardown path and crashes with java.lang.StackOverflowError. Because that is a java.lang.Error (not a RetriableException/ConnectException), it bypasses errors.tolerance and the retry machinery. The task ends in a terminal FAILED state and does not self-recover; it requires a manual task/connector restart.

The recursion is between three methods:

As400RpcConnection.connection() — when the cached AS400 is dead, calls close() at the top of its reconnect path.
As400RpcConnection.close() — calls retrieveJournal.cancelJob() before nulling this.as400 (the field that would break the cycle is only assigned at the end of close(), which is never reached).
RetrieveJournal.cancelJob() — when a job is set, calls config.as400().connection() to cancel it, re-entering connection() on the still-dead connection.
Cycle:

connection() [as400 dead] -> close() -> cancelJob() [job set]
-> connection() [as400 still dead] -> close() -> cancelJob() -> ... (StackOverflow)
Why it is intermittent: cancelJob() only proceeds past its job == null early-return when ibmiJob is set, and in RetrieveJournal.retrieveJournal(...) ibmiJob is non-null only across the blocking spc.run() call (set immediately before, cleared immediately after). So the recursion is hit only if the host disconnects while a connector is mid-spc.run(). If the host drops while idle between polls (ibmiJob == null), teardown completes cleanly. This matches the field symptom exactly: during a nightly backup, whichever connector(s) happen to be mid-retrieval at the instant the host goes down fail with StackOverflowError, while idle sibling connectors on the same worker survive; the failing connector varies night to night.

Representative trace (frames repeat until stack exhaustion):

Caused by: java.lang.StackOverflowError
at io.debezium.connector.db2as400.As400RpcConnection.close(As400RpcConnection.java:86)
at io.debezium.connector.db2as400.As400RpcConnection.connection(As400RpcConnection.java:125)
at io.debezium.connector.db2as400.As400RpcConnection.connection(As400RpcConnection.java:43)
at io.debezium.ibmi.db2.journal.retrieve.RetrieveJournal.cancelJob(RetrieveJournal.java:112)
at io.debezium.connector.db2as400.As400RpcConnection.close(As400RpcConnection.java:86)
... (repeats)`

---

**Do you see the same behaviour using the latest released Debezium version?**

_Ideally, also verify with the latest Alpha/Beta/CR version._

`We observed it on 3.4.0.Final. The recursive code path (close() calling cancelJob() before nulling the connection, and cancelJob() re-opening a connection via connection()) also appears present on the current main of debezium-connector-ibmi, so we do not expect a newer 3.5.x/3.6.x release to resolve it. We have not, however, run a controlled reproduction on the very latest release — noting that here so it can be confirmed.`

---

**Do you have the connector logs, ideally from start till finish?**

_You might be asked later to provide [DEBUG/TRACE](https://debezium.io/documentation/reference/stable/operations/logging.html) level log._

`We can provide Connect task logs showing the repeating close() → connection() → cancelJob() frames terminating in StackOverflowError and the task’s transition to FAILED. We can supply DEBUG/TRACE-level logs on request; the connection layer logs "create new as400 connection" and "Disconnecting" on each recursion iteration.`

---

**How to reproduce the issue using our [tutorial](https://github.com/debezium/debezium-examples/tree/main/tutorial) deployment?**

`The tutorial doesn’t include an IBM i source, so this can’t be reproduced with the standard tutorial images. To reproduce against a db2as400 source:

1. Run the db2as400 connector against an IBM i journal source and let it stream.
2. Interrupt connectivity to the IBM i host while a journal-retrieval RPC is in flight (i.e. during spc.run(), when ibmiJob is set) — e.g. shut down or network-isolate the host during active streaming. A nightly backup that stops the host is a reliable natural trigger.
3. Observe the task fail with StackOverflowError originating in the close() / connection() / cancelJob() cycle and remain in terminal FAILED.
4.
The timing condition (host lost during spc.run(), not while idle) is the key to reproducing it.`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.