kestra-io / kestra-io/plugin-jdbc

Dremio Query task errors out with `CONNECTION : SSL negotiation failed`

Open
#212 2 comments 0 reactions 0 assignees View on GitHub
area/plugin good first issue
Dominant language
Java
Stars
24
Forks
38
Avg merge
2d 3h
Merged PRs (30d)
22

Description

### Feature description

To reproduce, run:

```bash
docker run --pull=always --rm -it -p 28080:8080 --user=root -v /var/run/docker.sock:/var/run/docker.sock -e SECRET_DREMIO_TOKEN="SmFORmFrY1RUOUdwTlh4SkFjNDJSbXBFYUVCSS9NTi9NZVNzUGFyTlVUYTV0QUxZNytoSTVRcGI4ckVvZXc9PQ==" kestra/kestra:develop-full server local
```

and run this flow:

```yaml
id: dremio_sql_python
namespace: dremio

variables:
user: data.engineering.blog@gmail.com
project_id: ead79cc0-9e93-4d50-b364-77639a56d4a6

tasks:
- id: wdir
type: io.kestra.core.tasks.flows.WorkingDirectory
tasks:
- id: cloneRepository
type: io.kestra.plugin.git.Clone
url: https://github.com/dbt-labs/jaffle_shop
branch: main

- id: dbt
type: io.kestra.plugin.dbt.cli.DbtCLI
docker:
image: ghcr.io/kestra-io/dbt-dremio:latest
profiles: |
jaffle_shop:
outputs:
dev:
type: dremio
threads: 16
cloud_host: api.dremio.cloud
cloud_project_id: "{{ vars.project_id }}"
user: "{{ vars.user }}"
pat: "{{ secret('DREMIO_TOKEN') }}"
use_ssl: true
target: dev
commands:
- dbt build

- id: query
type: io.kestra.plugin.jdbc.dremio.Query
url: "jdbc:dremio:direct=sql.dremio.cloud:443;ssl=true;PROJECT_ID={{vars.project_id}};"
username: $token
password: "{{ secret('DREMIO_TOKEN') }}"
sql: SELECT * FROM "@{{ vars.user }}"."stg_customers";
store: true

- id: to_csv
type: io.kestra.plugin.serdes.csv.CsvWriter
from: "{{ outputs.query.uri }}"

- id: python
type: io.kestra.plugin.scripts.python.Script
warningOnStdErr: false
docker:
image: ghcr.io/kestra-io/polars:latest
inputFiles:
data.csv: "{{ outputs.to_csv.uri }}"
script: |
import polars as pl
df = pl.read_csv("data.csv")

print(df.glimpse())
```

full stack-trace:

```Java
java.sql.SQLException: Failure in connecting to Dremio: cdjd.com.dremio.exec.rpc.ConnectionFailedException: CONNECTION : SSL negotiation failed
at com.dremio.jdbc.impl.DremioExceptionMapper.map(DremioExceptionMapper.java:81)
at com.dremio.jdbc.impl.DremioConnectionImpl.(DremioConnectionImpl.java:115)
at com.dremio.jdbc.impl.DremioJdbc41Factory.newConnection(DremioJdbc41Factory.java:72)
at com.dremio.jdbc.impl.DremioFactory.newConnection(DremioFactory.java:67)
at cdjd.org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at com.dremio.jdbc.Driver.connect(Driver.java:70)
at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
at io.kestra.plugin.jdbc.JdbcConnectionInterface.connection(JdbcConnectionInterface.java:63)
at io.kestra.plugin.jdbc.AbstractJdbcQuery.run(AbstractJdbcQuery.java:77)
at io.kestra.plugin.jdbc.dremio.Query.run(Query.java:55)
at io.kestra.plugin.jdbc.dremio.Query.run(Query.java:20)
at io.kestra.core.runners.Worker$WorkerThread.run(Worker.java:727)
Caused by: cdjd.com.dremio.exec.rpc.ConnectionFailedException: CONNECTION : SSL negotiation failed
at cdjd.com.dremio.exec.rpc.ConnectionFailedException.mapException(ConnectionFailedException.java:31)
at cdjd.com.dremio.exec.client.DremioClient$FutureHandler.connectionFailed(DremioClient.java:830)
at cdjd.com.dremio.sabot.rpc.user.QueryResultHandler$ChannelClosedHandler.connectionFailed(QueryResultHandler.java:387)
at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.lambda$addNegotiator$0(BasicClient.java:341)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at cdjd.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at cdjd.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
at cdjd.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at cdjd.io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1804)
at cdjd.io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1773)
at cdjd.io.netty.handler.ssl.SslHandler.handshake(SslHandler.java:1996)
at cdjd.io.netty.handler.ssl.SslHandler.startHandshakeProcessing(SslHandler.java:1914)
at cdjd.io.netty.handler.ssl.SslHandler.handlerAdded(SslHandler.java:1903)
at cdjd.io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:971)
at cdjd.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
at cdjd.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:181)
at cdjd.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:152)
at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.addNegotiator(BasicClient.java:350)
at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.operationComplete(BasicClient.java:291)
at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.operationComplete(BasicClient.java:272)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at cdjd.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at cdjd.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
at cdjd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
at cdjd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
at cdjd.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:300)
at cdjd.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:335)
at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at cdjd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at cdjd.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at cdjd.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: cdjd.com.dremio.exec.rpc.RpcException: CONNECTION : SSL negotiation failed
at cdjd.com.dremio.exec.client.DremioClient$FutureHandler.connectionFailed(DremioClient.java:826)
... 37 common frames omitted
Caused by: cdjd.com.dremio.exec.rpc.RpcException: SSL negotiation failed
at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.lambda$addNegotiator$0(BasicClient.java:342)
... 35 common frames omitted
Caused by: java.lang.UnsupportedOperationException: Allocator doesn't support heap-based memory.
at cdjd.org.apache.arrow.memory.ArrowByteBufAllocator.fail(ArrowByteBufAllocator.java:158)
at cdjd.org.apache.arrow.memory.ArrowByteBufAllocator.heapBuffer(ArrowByteBufAllocator.java:129)
at cdjd.io.netty.handler.ssl.SslHandler$SslEngineType$3.allocateWrapBuffer(SslHandler.java:311)
at cdjd.io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:2162)
at cdjd.io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:941)
at cdjd.io.netty.handler.ssl.SslHandler.handshake(SslHandler.java:1994)
... 26 common frames omitted
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the supplied flow and reading io.kestra.plugin.jdbc.dremio.Query and io.kestra.plugin.jdbc.JdbcConnectionInterface.connection. Trace the Dremio JDBC and Arrow allocator failure shown in the stack trace. Done means the Dremio query connects successfully and the flow can continue to CSV and Python tasks without the SSL negotiation error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.