kestra-io / kestra-io/plugin-jdbc
Dremio Query task errors out with `CONNECTION : SSL negotiation failed`
- Dominant language
- Java
- Stars
- 24
- Forks
- 38
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 22
Description
### Feature description
To reproduce, run:
```bash
docker run --pull=always --rm -it -p 28080:8080 --user=root -v /var/run/docker.sock:/var/run/docker.sock -e SECRET_DREMIO_TOKEN="SmFORmFrY1RUOUdwTlh4SkFjNDJSbXBFYUVCSS9NTi9NZVNzUGFyTlVUYTV0QUxZNytoSTVRcGI4ckVvZXc9PQ==" kestra/kestra:develop-full server local
```
and run this flow:
```yaml
id: dremio_sql_python
namespace: dremio
variables:
user: data.engineering.blog@gmail.com
project_id: ead79cc0-9e93-4d50-b364-77639a56d4a6
tasks:
- id: wdir
type: io.kestra.core.tasks.flows.WorkingDirectory
tasks:
- id: cloneRepository
type: io.kestra.plugin.git.Clone
url: https://github.com/dbt-labs/jaffle_shop
branch: main
- id: dbt
type: io.kestra.plugin.dbt.cli.DbtCLI
docker:
image: ghcr.io/kestra-io/dbt-dremio:latest
profiles: |
jaffle_shop:
outputs:
dev:
type: dremio
threads: 16
cloud_host: api.dremio.cloud
cloud_project_id: "{{ vars.project_id }}"
user: "{{ vars.user }}"
pat: "{{ secret('DREMIO_TOKEN') }}"
use_ssl: true
target: dev
commands:
- dbt build
- id: query
type: io.kestra.plugin.jdbc.dremio.Query
url: "jdbc:dremio:direct=sql.dremio.cloud:443;ssl=true;PROJECT_ID={{vars.project_id}};"
username: $token
password: "{{ secret('DREMIO_TOKEN') }}"
sql: SELECT * FROM "@{{ vars.user }}"."stg_customers";
store: true
- id: to_csv
type: io.kestra.plugin.serdes.csv.CsvWriter
from: "{{ outputs.query.uri }}"
- id: python
type: io.kestra.plugin.scripts.python.Script
warningOnStdErr: false
docker:
image: ghcr.io/kestra-io/polars:latest
inputFiles:
data.csv: "{{ outputs.to_csv.uri }}"
script: |
import polars as pl
df = pl.read_csv("data.csv")
print(df.glimpse())
```
full stack-trace:
```Java
java.sql.SQLException: Failure in connecting to Dremio: cdjd.com.dremio.exec.rpc.ConnectionFailedException: CONNECTION : SSL negotiation failed
at com.dremio.jdbc.impl.DremioExceptionMapper.map(DremioExceptionMapper.java:81)
at com.dremio.jdbc.impl.DremioConnectionImpl.(DremioConnectionImpl.java:115)
at com.dremio.jdbc.impl.DremioJdbc41Factory.newConnection(DremioJdbc41Factory.java:72)
at com.dremio.jdbc.impl.DremioFactory.newConnection(DremioFactory.java:67)
at cdjd.org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at com.dremio.jdbc.Driver.connect(Driver.java:70)
at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
at io.kestra.plugin.jdbc.JdbcConnectionInterface.connection(JdbcConnectionInterface.java:63)
at io.kestra.plugin.jdbc.AbstractJdbcQuery.run(AbstractJdbcQuery.java:77)
at io.kestra.plugin.jdbc.dremio.Query.run(Query.java:55)
at io.kestra.plugin.jdbc.dremio.Query.run(Query.java:20)
at io.kestra.core.runners.Worker$WorkerThread.run(Worker.java:727)
Caused by: cdjd.com.dremio.exec.rpc.ConnectionFailedException: CONNECTION : SSL negotiation failed
at cdjd.com.dremio.exec.rpc.ConnectionFailedException.mapException(ConnectionFailedException.java:31)
at cdjd.com.dremio.exec.client.DremioClient$FutureHandler.connectionFailed(DremioClient.java:830)
at cdjd.com.dremio.sabot.rpc.user.QueryResultHandler$ChannelClosedHandler.connectionFailed(QueryResultHandler.java:387)
at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.lambda$addNegotiator$0(BasicClient.java:341)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at cdjd.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at cdjd.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
at cdjd.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at cdjd.io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1804)
at cdjd.io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1773)
at cdjd.io.netty.handler.ssl.SslHandler.handshake(SslHandler.java:1996)
at cdjd.io.netty.handler.ssl.SslHandler.startHandshakeProcessing(SslHandler.java:1914)
at cdjd.io.netty.handler.ssl.SslHandler.handlerAdded(SslHandler.java:1903)
at cdjd.io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:971)
at cdjd.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
at cdjd.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:181)
at cdjd.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:152)
at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.addNegotiator(BasicClient.java:350)
at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.operationComplete(BasicClient.java:291)
at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.operationComplete(BasicClient.java:272)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at cdjd.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at cdjd.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
at cdjd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
at cdjd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
at cdjd.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:300)
at cdjd.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:335)
at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at cdjd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at cdjd.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at cdjd.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: cdjd.com.dremio.exec.rpc.RpcException: CONNECTION : SSL negotiation failed
at cdjd.com.dremio.exec.client.DremioClient$FutureHandler.connectionFailed(DremioClient.java:826)
... 37 common frames omitted
Caused by: cdjd.com.dremio.exec.rpc.RpcException: SSL negotiation failed
at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.lambda$addNegotiator$0(BasicClient.java:342)
... 35 common frames omitted
Caused by: java.lang.UnsupportedOperationException: Allocator doesn't support heap-based memory.
at cdjd.org.apache.arrow.memory.ArrowByteBufAllocator.fail(ArrowByteBufAllocator.java:158)
at cdjd.org.apache.arrow.memory.ArrowByteBufAllocator.heapBuffer(ArrowByteBufAllocator.java:129)
at cdjd.io.netty.handler.ssl.SslHandler$SslEngineType$3.allocateWrapBuffer(SslHandler.java:311)
at cdjd.io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:2162)
at cdjd.io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:941)
at cdjd.io.netty.handler.ssl.SslHandler.handshake(SslHandler.java:1994)
... 26 common frames omitted
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the supplied flow and reading io.kestra.plugin.jdbc.dremio.Query and io.kestra.plugin.jdbc.JdbcConnectionInterface.connection. Trace the Dremio JDBC and Arrow allocator failure shown in the stack trace. Done means the Dremio query connects successfully and the flow can continue to CSV and Python tasks without the SSL negotiation error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100