ClickHouse / ClickHouse/clickhouse-java

Enhance insert() API to support SELECT with input() for computed columns v2

Offen
#2,726 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
enhancement
Vorherrschende Sprache
Java
Sterne
1.6k
Forks
636
Ø Merge
2 T. 23 Std.
Gemergte PRs (30 T.)
29

Beschreibung

### Current Limitation
The `insert()` method only supports direct table insertion, not SELECT statements
with transformations.

### Desired Enhancement
Allow `insert()` to accept SELECT statements with input() function:
```java
client.insert()
.table("target_table")
.query("SELECT col1, col2, col1 + col2 as computed FROM input('col1 Int32, col2 Int32')")
.data(inputStream)
.format(ClickHouseFormat.CSV)
.send();
```

Or alternative syntax:
```java
client.insert("INSERT INTO target_table SELECT col1, col2, col1 + col2 as computed FROM input('col1 Int32, col2 Int32')")
.data(inputStream)
.format(ClickHouseFormat.CSV)
.send();
```

### Use Case
We need to stream data with computed columns during insertion:
- Stream large datasets efficiently
- Apply transformations/computed columns on the fly
- Leverage ClickHouse's input() function capabilities
- Maintain the performance benefits of the streaming insert() API

### Business Value
This is a common pattern in ClickHouse for ETL workloads where data needs
transformation during insertion without materializing intermediate results.

### Current Workaround (Working but Undocumented)

We're successfully using this pattern in production:
```java
String loadQuery = tableName + "(" + headers + ")" +
" select " + parametersQuery + "* from input('" + csvHeadersStructure + "')";

try (InputStream inputStream = Files.newInputStream(Paths.get(filepath))) {
InsertResponse response = client.insert(loadQuery, inputStream, ClickHouseFormat.CSVWithNames)
.get(1200, TimeUnit.SECONDS);
}
```

**This works but:**
- Not documented in official API docs
- Unclear if this is intended behavior or might break in future versions
- No IDE autocomplete/type safety for this pattern

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start at the Java client's insert() API entry points and existing handling of query strings, then compare the builder and overload forms with the working input() workaround described here. Define and test the supported SELECT/input() syntax with streaming data and computed columns, and document whether the existing pattern is intentional.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
api, backend-api-design
Issue-Typ
Feature
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
48/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.