ClickHouse / ClickHouse/clickhouse-java
Insert function is not working
- Dominant language
- Java
- Stars
- 1.6k
- Forks
- 636
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 29
Description
### I am not able to insert the CSV data into the existing table in click house. using clickhouse-java
I run the below code but it throws the errro:
```
static long insert(ClickHouseNode server, String table) throws ClickHouseException {
try (ClickHouseClient client = ClickHouseClient.newInstance(server.getProtocol())) {
ClickHouseRequest.Mutation request = client.connect(server).write().table(table)
.format(ClickHouseFormat.RowBinary);
ClickHouseConfig config = request.getConfig();
CompletableFuture future;
// back-pressuring is not supported, you can adjust the first two arguments
try (ClickHousePipedOutputStream stream = ClickHouseDataStreamFactory.getInstance()
.createPipedOutputStream(config, (Runnable) null)) {
// in async mode, which is default, execution happens in a worker thread
future = request.data(stream.getInputStream()).execute();
// writing happens in main thread
for (int i = 0; i < 10_000; i++) {
BinaryStreamUtils.writeString(stream, String.valueOf(i % 16));
BinaryStreamUtils.writeNonNull(stream);
BinaryStreamUtils.writeString(stream, UUID.randomUUID().toString());
}
}
// response should be always closed
try (ClickHouseResponse response = future.get()) {
ClickHouseResponseSummary summary = response.getSummary();
return summary.getWrittenRows();
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw ClickHouseException.forCancellation(e, server);
} catch (ExecutionException | IOException e) {
throw ClickHouseException.of(e, server);
}
}
```
### Error
```
java.util.concurrent.ExecutionException: com.clickhouse.client.ClickHouseException: Code: 33. DB::Exception: Cannot read all data. Bytes read: 2. Bytes expected: 4.: (at row 14272)
: While executing BinaryRowInputFormat. (CANNOT_READ_ALL_DATA) (version 23.6.2.18 (official build))
, server ClickHouseNode [uri=http://localhost:8123/ShabirBhat_project]@-2130346309
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at com.datoin.ch.CHclientTest.insert(CHclientTest.java:77)
at com.datoin.ch.CHclientTest.main(CHclientTest.java:40)
Caused by: com.clickhouse.client.ClickHouseException: Code: 33. DB::Exception: Cannot read all data. Bytes read: 2. Bytes expected: 4.: (at row 14272)
: While executing BinaryRowInputFormat. (CANNOT_READ_ALL_DATA) (version 23.6.2.18 (official build))
, server ClickHouseNode [uri=http://localhost:8123/ShabirBhat_project]@-2130346309
at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168)
at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: Code: 33. DB::Exception: Cannot read all data. Bytes read: 2. Bytes expected: 4.: (at row 14272)
: While executing BinaryRowInputFormat. (CANNOT_READ_ALL_DATA) (version 23.6.2.18 (official build))
at com.clickhouse.client.http.HttpUrlConnectionImpl.checkResponse(HttpUrlConnectionImpl.java:184)
at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:227)
at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124)
at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161)
at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273)
... 4 more
Process finished with exit code 0
```
# I tried to replace the below code with my csv data
```
// writing happens in main thread
for (int i = 0; i < 10_000; i++) {
BinaryStreamUtils.writeString(stream, String.valueOf(i % 16));
BinaryStreamUtils.writeNonNull(stream);
BinaryStreamUtils.writeString(stream, UUID.randomUUID().toString());
}
Replace with :
BufferedReader reader = new BufferedReader(new FileReader(csvFilePath));
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
BinaryStreamUtils.writeString(stream, line);
}
```
Contributor guide
Research direction
Start at the insert method in the reported CHclientTest.java stack trace and review the ClickHouseRequest.Mutation setup using RowBinary. Compare the original BinaryStreamUtils writes with the later CSV line-writing attempt, and check the target table schema and expected input format. Done means the example inserts the CSV data without the CANNOT_READ_ALL_DATA error and reports the written rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100