ClickHouse / ClickHouse/clickhouse-java
java.sql.BatchUpdateException: Broken pipe (Write failed), server ClickHouseNode
- Langage dominant
- Java
- Étoiles
- 1.6k
- Forks
- 636
- Merge moyen
- 2 j 23 h
- PR mergées (30 j)
- 29
Description
I write a Scala code using Spark to push data from a Hive table to ClickHouse. When I use the clickhouse-jdbc version 0.6.0, I set the following parameters.
```
properties.setProperty("http_connection_provider", HttpConnectionProvider.APACHE_HTTP_CLIENT.name());
properties.setProperty("socket_ip_tos", "32");
```
There is no problem when predicting 5kw data , but an exception is thrown when it becomes 5e data. threw an error as follows:
```
24/04/25 15:05:53 [task-result-getter-0] WARN TaskSetManager: Lost task 67.0 in stage 1.0 (TID 690, 11.11.10.76, executor 60): java.sql.BatchUpdateException: Broken pipe (Write failed), server ClickHouseNode [uri=http://sq02-ch-000119-clickhouse-17-1.local:8623/default, options={db=ge_order,socket_ip_tos=32,http_connection_provider=APACHE_HTTP_CLIENT}]@290467083
at com.clickhouse.jdbc.SqlExceptionUtils.batchUpdateError(SqlExceptionUtils.java:107)
at com.clickhouse.jdbc.internal.InputBasedPreparedStatement.executeAny(InputBasedPreparedStatement.java:154)
at com.clickhouse.jdbc.internal.AbstractPreparedStatement.executeLargeBatch(AbstractPreparedStatement.java:85)
at com.clickhouse.jdbc.internal.ClickHouseStatementImpl.executeBatch(ClickHouseStatementImpl.java:752)
at com.jd.clickhouse.spark.DataFrameExt$$anonfun$13$$anonfun$apply$23.apply(DataFrameExt.scala:513)
at com.jd.clickhouse.spark.DataFrameExt$$anonfun$13$$anonfun$apply$23.apply(DataFrameExt.scala:467)
```
When I adjust it to:
```
properties.setProperty("http_connection_provider", HttpConnectionProvider.HTTP_URL_CONNECTION.name())
```
threw an error as follows:
```
WARN TaskSetManager: Lost task 91.0 in stage 1.0 (TID 717, 10.198.62.132, executor 116): java.sql.BatchUpdateException: Error writing request body to server, server ClickHouseNode [uri=http://sq02-ch-000119-clickhouse-23-1.local:8623/default, options={db=ge_order,socket_ip_tos=32,http_connection_provider=HTTP_URL_CONNECTION}]@665046067
at com.clickhouse.jdbc.SqlExceptionUtils.batchUpdateError(SqlExceptionUtils.java:107)
at com.clickhouse.jdbc.internal.InputBasedPreparedStatement.executeAny(InputBasedPreparedStatement.java:154)
at com.clickhouse.jdbc.internal.AbstractPreparedStatement.executeLargeBatch(AbstractPreparedStatement.java:85)
at com.clickhouse.jdbc.internal.ClickHouseStatementImpl.executeBatch(ClickHouseStatementImpl.java:752)
at com.jd.clickhouse.spark.DataFrameExt$$anonfun$13$$anonfun$apply$23.apply(DataFrameExt.scala:513)
```
When I use the ru.yandex.clickhouse 0.1.34 ,The same problem will not occur.
The versions of ClickHouse, Spark version are all consistent.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par les appels en échec dans DataFrameExt.scala aux lignes 467 et 513, puis examinez InputBasedPreparedStatement.executeAny de clickhouse-jdbc ainsi que les deux fournisseurs de connexions HTTP. Reproduisez le gros lot Spark/Hive avec clickhouse-jdbc 0.6.0 et comparez-le à ru.yandex.clickhouse 0.1.34 ; le travail est terminé lorsqu’une cause reproductible et une correction vérifiée ou une configuration prise en charge pour l’échec broken pipe ont été identifiées.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- clickhouse, java, scala, spark
- Domaine
- data-engineering, databases
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- Calme
- Clarté
- À clarifier
- Accessibilité débutants
- 40/100