ClickHouse / ClickHouse/clickhouse-rs
clickhouse-rust: `Network("channel closed")` on bulk insert (3000+ rows, 155 columns)
- Dominant language
- Rust
- Stars
- 559
- Forks
- 172
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 3
Description
# clickhouse-rust: `Network("channel closed")` on bulk insert (3000+ rows, 155 columns)
**Crate:** `clickhouse` v0.15.0
**Repository:** https://github.com/ClickHouse/rust-clickhouse
**ClickHouse Server:** 24.x (direct connection, no proxy)
## Problem
Using the `Inserter` API to insert ~3000 rows with 155 columns reliably fails with `Network("channel closed")`. The same data inserts successfully via plain HTTP POST (`reqwest`, 0.1s).
## Reproduction
```rust
use std::collections::HashMap;
let mut inserter = client
.inserter::("cn_stock_order")
.await?
.with_setting("send_timeout", "600")
.with_setting("max_insert_block_size", "800")
.end()
.await?;
for (&sid, features) in &features_map {
inserter.write(&row).await?;
}
inserter.end().await?; // fails here: Network("channel closed")
```
**Row struct:** 157 fields (date `Date`, symbol `String`, 155 `Float64` columns).
**Server settings observed:**
- `http_send_timeout` = 30s (socket-level, not overridable by client)
- `http_receive_timeout` = 30s
- `send_timeout` = 300s
## Root Cause Analysis
The crate's `Inserter` uses **chunked transfer encoding** with an internal spawned tokio task that streams rows to the server. Between chunks, idle gaps form on the connection. When a gap exceeds the server's `http_send_timeout` (30s), the server closes the socket.
Key observations:
1. `with_setting("send_timeout", "600")` has no effect — `http_send_timeout` is enforced at the socket layer and cannot be overridden by client-side settings.
2. `with_setting("max_insert_block_size", "800")` with periodic `end()` helps smaller batches but still fails on ~3000 rows.
3. SSE dataset (~3000+ rows) times out consistently. SZE dataset (~3077 rows) sometimes succeeds, suggesting it's a timing-dependent race.
4. Plain HTTP POST with the same data (full body, single request) completes in 0.1s — the server can accept the data instantly when sent without chunking.
## Expected Behavior
Inserting 3000 rows with 155 columns (~3 MB uncompressed TSV) should succeed without socket timeouts, as the ClickHouse server accepts the same data instantly via direct HTTP POST.
## Workaround
Use `reqwest` with plain HTTP POST:
```rust
let query = "INSERT INTO cn_stock_order FORMAT TabSeparated";
let url = format!("http://host:8123/?user=default&query={}", urlencoding(query));
reqwest::Client::new()
.post(&url)
.body(tsv_body)
.send()
.await?;
```
## Environment
- `clickhouse` crate: 0.15.0 (features: `chrono`, `inserter`, `zstd`)
- ClickHouse server: 24.x on 172.16.210.220:8123
- Rust: edition 2024, tokio 1.x (full features)
- No nginx/reverse proxy (direct ClickHouse connection confirmed)
Contributor guide
Research direction
Start at the Inserter API and its internal spawned task that uses chunked transfer encoding, then reproduce the 3000-row, 155-column case against ClickHouse 24.x. Done means the bulk insert completes without Network("channel closed") or a socket timeout; compare it with the plain reqwest POST behavior described in the report.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, rust
- Domain
- databases, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100