apache / apache/seatunnel

[Bug] [Sink Clickhouse] Cannot set null to non-nullable column

Open
#5,569 3 comments 0 reactions 0 assignees View on GitHub
bug Waiting for users feedback
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 13h
Merged PRs (30d)
203

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

hive sink ck, exception: Cannot set null to non-nullable colum

### SeaTunnel Version

v2.3.3

### SeaTunnel Config

```conf
env {
execution.parallelism = 1
job.mode = "BATCH"
}

source {
Hive {
table_name = "hive_db.table1"
sql = "select * from hive_db.table1"
result_table_name = "res1"
metastore_uri = "******:9083"
}
}

transform {
}

sink {
Clickhouse {
host = "******:80"
database = "ck_db"
table = "table1_all"
username = "user-clusters"
password = "******"
}
}
```

### Running Command

```shell
./bin/start-seatunnel-spark-3-connector-v2.sh \
--master local[4] \
--deploy-mode client \
--config ./config/seatunnel.streaming.conf.template
```

### Error Exception

```log
Caused by: java.sql.SQLException: Cannot set null to non-nullable column #48 [last_order_creation_date_with_outbound String]
at com.clickhouse.jdbc.SqlExceptionUtils.clientError(SqlExceptionUtils.java:73)
at com.clickhouse.jdbc.internal.InputBasedPreparedStatement.addBatch(InputBasedPreparedStatement.java:328)
at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.FieldNamedPreparedStatement.addBatch(FieldNamedPreparedStatement.java:371)
at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.SimpleBatchStatementExecutor.addToBatch(SimpleBatchStatementExecutor.java:43)
at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.BufferedBatchStatementExecutor.executeBatch(BufferedBatchStatementExecutor.java:51)
at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.flush(ClickhouseSinkWriter.java:124)
... 16 more
```

### Zeta or Flink or Spark Version

Spark 3.2

### Java or Scala Version

java8

### Screenshots

image

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Hive-to-ClickHouse batch with the provided configuration and inspect the failing column reported in the exception. Read FieldNamedPreparedStatement.java, SimpleBatchStatementExecutor.java, and ClickhouseSinkWriter.java around the stack-trace locations. Done means the reported input no longer fails on the non-nullable column, with the expected behavior covered by an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.