[Bug] [Sink Clickhouse] Cannot set null to non-nullable column
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 203
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
hive sink ck, exception: Cannot set null to non-nullable colum
### SeaTunnel Version
v2.3.3
### SeaTunnel Config
```conf
env {
execution.parallelism = 1
job.mode = "BATCH"
}
source {
Hive {
table_name = "hive_db.table1"
sql = "select * from hive_db.table1"
result_table_name = "res1"
metastore_uri = "******:9083"
}
}
transform {
}
sink {
Clickhouse {
host = "******:80"
database = "ck_db"
table = "table1_all"
username = "user-clusters"
password = "******"
}
}
```
### Running Command
```shell
./bin/start-seatunnel-spark-3-connector-v2.sh \
--master local[4] \
--deploy-mode client \
--config ./config/seatunnel.streaming.conf.template
```
### Error Exception
```log
Caused by: java.sql.SQLException: Cannot set null to non-nullable column #48 [last_order_creation_date_with_outbound String]
at com.clickhouse.jdbc.SqlExceptionUtils.clientError(SqlExceptionUtils.java:73)
at com.clickhouse.jdbc.internal.InputBasedPreparedStatement.addBatch(InputBasedPreparedStatement.java:328)
at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.FieldNamedPreparedStatement.addBatch(FieldNamedPreparedStatement.java:371)
at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.SimpleBatchStatementExecutor.addToBatch(SimpleBatchStatementExecutor.java:43)
at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.BufferedBatchStatementExecutor.executeBatch(BufferedBatchStatementExecutor.java:51)
at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.flush(ClickhouseSinkWriter.java:124)
... 16 more
```
### Zeta or Flink or Spark Version
Spark 3.2
### Java or Scala Version
java8
### Screenshots
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Hive-to-ClickHouse batch with the provided configuration and inspect the failing column reported in the exception. Read FieldNamedPreparedStatement.java, SimpleBatchStatementExecutor.java, and ClickhouseSinkWriter.java around the stack-trace locations. Done means the reported input no longer fails on the non-nullable column, with the expected behavior covered by an appropriate regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, java, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100