ClickHouse / ClickHouse/clickhouse-java
Enhance insert() API to support SELECT with input() for computed columns v2
- Vorherrschende Sprache
- Java
- Sterne
- 1.6k
- Forks
- 636
- Ø Merge
- 2 T. 23 Std.
- Gemergte PRs (30 T.)
- 29
Beschreibung
### Current Limitation
The `insert()` method only supports direct table insertion, not SELECT statements
with transformations.
### Desired Enhancement
Allow `insert()` to accept SELECT statements with input() function:
```java
client.insert()
.table("target_table")
.query("SELECT col1, col2, col1 + col2 as computed FROM input('col1 Int32, col2 Int32')")
.data(inputStream)
.format(ClickHouseFormat.CSV)
.send();
```
Or alternative syntax:
```java
client.insert("INSERT INTO target_table SELECT col1, col2, col1 + col2 as computed FROM input('col1 Int32, col2 Int32')")
.data(inputStream)
.format(ClickHouseFormat.CSV)
.send();
```
### Use Case
We need to stream data with computed columns during insertion:
- Stream large datasets efficiently
- Apply transformations/computed columns on the fly
- Leverage ClickHouse's input() function capabilities
- Maintain the performance benefits of the streaming insert() API
### Business Value
This is a common pattern in ClickHouse for ETL workloads where data needs
transformation during insertion without materializing intermediate results.
### Current Workaround (Working but Undocumented)
We're successfully using this pattern in production:
```java
String loadQuery = tableName + "(" + headers + ")" +
" select " + parametersQuery + "* from input('" + csvHeadersStructure + "')";
try (InputStream inputStream = Files.newInputStream(Paths.get(filepath))) {
InsertResponse response = client.insert(loadQuery, inputStream, ClickHouseFormat.CSVWithNames)
.get(1200, TimeUnit.SECONDS);
}
```
**This works but:**
- Not documented in official API docs
- Unclear if this is intended behavior or might break in future versions
- No IDE autocomplete/type safety for this pattern
Beitragsleitfaden
Rechercherichtung
Start at the Java client's insert() API entry points and existing handling of query strings, then compare the builder and overload forms with the working input() workaround described here. Define and test the supported SELECT/input() syntax with streaming data and computed columns, and document whether the existing pattern is intentional.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- java
- Bereich
- api, backend-api-design
- Issue-Typ
- Feature
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 48/100