airbytehq / airbytehq/airbyte

[source-clickhouse] MEMORY_LIMIT_EXCEEDED on large S3-backed table due to unbatched full table scan

Abierto
#76,178 3 comentarios 0 reacciones 0 asignados Ver en GitHub
area/connectors autoteam community connectors/source/clickhouse needs-triage team/extensibility type/bug
Lenguaje dominante
Python
Estrellas
22.1k
Forks
5.3k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

### Connector Name

source-clickhouse (ClickHouse v0.3.0)

### Connector Version

v0.3.0

### What step the error happened?

During the sync

### Relevant information

We are syncing a large ClickHouse table (`default.usage_events`) using the ClickHouse source connector. The sync consistently fails with MEMORY_LIMIT_EXCEEDED before completing.

**Environment:**
- ClickHouse version: 25.1.2.1.1426
- Table: default.usage_events
- Disk type: s3WithKeeperDisk (S3-backed storage)
- Connector version: (same as above)
- Sync mode: Incremental

**Error:**
Code 241: MEMORY_LIMIT_EXCEEDED — query killed by OvercommitTracker after reaching the 21.60 GiB memory limit while reading column `dimensions` from table `default.usage_events` (part: all_0_349248029_100052_382803101, starting at mark 59080).

**Root cause:**
The connector issues a full unbatched table scan with no chunking by time range, cursor value range, or row count. For large S3-backed tables, this exhausts instance memory before the query completes. The `dimensions` column (avg ~29.7 chars/row) is the primary memory driver.

**What we've tried:**
- Attempted to pass `max_memory_usage` and related settings via `jdbc_url_params` — confirmed unreliable with the current JDBC driver (per Airbyte support, PR #75432 is in progress to fix this)
- Confirmed with Airbyte support that the connector does not support batched/chunked reads

**Feature requests / expected behavior:**
1. Support chunked reads — split queries by cursor value range or row count (similar to how other JDBC-based connectors handle large tables)
2. Reliable passthrough of ClickHouse server settings (`max_memory_usage`, `max_rows_to_read`, `max_bytes_to_read`) via `jdbc_url_params` once PR #75432 lands

### Relevant log output

```shell

```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11927

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.