ClickHouse / ClickHouse/clickhouse-java

Enhance insert() API to support SELECT with input() for computed columns v2

Open
#2,726 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.6k
Forks
636
Avg merge
2d 23h
Merged PRs (30d)
29

Description

### Current Limitation
The `insert()` method only supports direct table insertion, not SELECT statements
with transformations.

### Desired Enhancement
Allow `insert()` to accept SELECT statements with input() function:
```java
client.insert()
.table("target_table")
.query("SELECT col1, col2, col1 + col2 as computed FROM input('col1 Int32, col2 Int32')")
.data(inputStream)
.format(ClickHouseFormat.CSV)
.send();
```

Or alternative syntax:
```java
client.insert("INSERT INTO target_table SELECT col1, col2, col1 + col2 as computed FROM input('col1 Int32, col2 Int32')")
.data(inputStream)
.format(ClickHouseFormat.CSV)
.send();
```

### Use Case
We need to stream data with computed columns during insertion:
- Stream large datasets efficiently
- Apply transformations/computed columns on the fly
- Leverage ClickHouse's input() function capabilities
- Maintain the performance benefits of the streaming insert() API

### Business Value
This is a common pattern in ClickHouse for ETL workloads where data needs
transformation during insertion without materializing intermediate results.

### Current Workaround (Working but Undocumented)

We're successfully using this pattern in production:
```java
String loadQuery = tableName + "(" + headers + ")" +
" select " + parametersQuery + "* from input('" + csvHeadersStructure + "')";

try (InputStream inputStream = Files.newInputStream(Paths.get(filepath))) {
InsertResponse response = client.insert(loadQuery, inputStream, ClickHouseFormat.CSVWithNames)
.get(1200, TimeUnit.SECONDS);
}
```

**This works but:**
- Not documented in official API docs
- Unclear if this is intended behavior or might break in future versions
- No IDE autocomplete/type safety for this pattern

Contributor guide

Open the contributing guide

Research direction

Start at the Java client's insert() API entry points and existing handling of query strings, then compare the builder and overload forms with the working input() workaround described here. Define and test the supported SELECT/input() syntax with streaming data and computed columns, and document whether the existing pattern is intentional.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.