aws / aws/amazon-redshift-jdbc-driver
Getting java.base/java.util.ArrayDeque.removeFirst(ArrayDeque.java:363) exception intermittently
- Dominant language
- Java
- Stars
- 71
- Forks
- 42
- PR merge metrics
- No merged PRs in 30d
Description
## Driver version
2.1.0.26
## Redshift version
PostgreSQL 8.0.2 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.4.2 20041017 (Red Hat 3.4.2-6.fc3), Redshift 1.0.68555
## Client Operating System
linux/amd64
## Table schema
## Problem description
We use a [Kafka Connect Docker Image](https://hub.docker.com/layers/confluentinc/cp-kafka-connect/7.4.1/images/sha256-b2e3a877fe6e6aac12abed3333176b939282135b95640b8d30798fa5e38e2974?context=explore) deployed over ECS with a [JDBC sink connector](https://github.com/confluentinc/kafka-connect-jdbc/tree/v10.6.0) `v10.6.0` and the Redshift JDBC driver to transfer data from MSK into a Redshift Serverless instance.
This intermittently fails with the following stack trace:
```
Caused by: java.util.NoSuchElementException
at java.base/java.util.ArrayDeque.removeFirst(ArrayDeque.java:363)
at com.amazon.redshift.core.v3.QueryExecutorImpl.processResultsOnThread(QueryExecutorImpl.java:2368)
at com.amazon.redshift.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1881)
at com.amazon.redshift.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1873)
at com.amazon.redshift.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:595)
at com.amazon.redshift.jdbc.RedshiftStatementImpl.internalExecuteBatch(RedshiftStatementImpl.java:988)
at com.amazon.redshift.jdbc.RedshiftStatementImpl.executeBatch(RedshiftStatementImpl.java:1016)
at com.amazon.redshift.jdbc.RedshiftPreparedStatement.executeBatch(RedshiftPreparedStatement.java:1765)
at io.confluent.connect.jdbc.sink.BufferedRecords.executeUpdates(BufferedRecords.java:196)
at io.confluent.connect.jdbc.sink.BufferedRecords.flush(BufferedRecords.java:186)
at io.confluent.connect.jdbc.sink.JdbcDbWriter.write(JdbcDbWriter.java:80)
at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:84)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:587)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:336)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:237)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:206)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:202)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:257)
at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:181)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```
when an `INSERT` query has run after a long time. Redshift Serverless logs suggest that the query succeeds and then the connector fails.
Usually restarting the task fixes the issue but since it doesn't throw a `java.sql.SQLException`, we can't automate the restarts.
## Reproduction code
The following configurations are set on the connector, all other values are left as default:
```
connection.url: jdbc:redshift://host:port/dbname
connection.user: username
connection.password: password
table.name.format: tablename
```
The connection parameters are fetched [here](https://github.com/confluentinc/kafka-connect-jdbc/blob/aa8466c30b09d3d12f4ca502b16f3c19ac7b2593/src/main/java/io/confluent/connect/jdbc/source/JdbcSourceConnectorConfig.java#L66) and passed on to the driver [here](https://github.com/confluentinc/kafka-connect-jdbc/blob/aa8466c30b09d3d12f4ca502b16f3c19ac7b2593/src/main/java/io/confluent/connect/jdbc/dialect/GenericDatabaseDialect.java#L245)
Contributor guide
Research direction
Start at com.amazon.redshift.core.v3.QueryExecutorImpl.processResultsOnThread and trace the batch path through RedshiftStatementImpl.executeBatch and BufferedRecords.executeUpdates. Run the reported Kafka Connect JDBC sink configuration with Redshift Serverless, focusing on an INSERT after a long idle period. Done means the intermittent failure is reproduced and its handling or cause is verified with a regression test or documented diagnostic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, java, kafka
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100