ClickHouse / ClickHouse/clickhouse-rs

clickhouse-rust: `Network("channel closed")` on bulk insert (3000+ rows, 155 columns)

Open
#437 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
559
Forks
172
Avg merge
1d 19h
Merged PRs (30d)
3

Description

# clickhouse-rust: `Network("channel closed")` on bulk insert (3000+ rows, 155 columns)

**Crate:** `clickhouse` v0.15.0
**Repository:** https://github.com/ClickHouse/rust-clickhouse
**ClickHouse Server:** 24.x (direct connection, no proxy)

## Problem

Using the `Inserter` API to insert ~3000 rows with 155 columns reliably fails with `Network("channel closed")`. The same data inserts successfully via plain HTTP POST (`reqwest`, 0.1s).

## Reproduction

```rust
use std::collections::HashMap;

let mut inserter = client
.inserter::("cn_stock_order")
.await?
.with_setting("send_timeout", "600")
.with_setting("max_insert_block_size", "800")
.end()
.await?;

for (&sid, features) in &features_map {
inserter.write(&row).await?;
}
inserter.end().await?; // fails here: Network("channel closed")
```

**Row struct:** 157 fields (date `Date`, symbol `String`, 155 `Float64` columns).

**Server settings observed:**
- `http_send_timeout` = 30s (socket-level, not overridable by client)
- `http_receive_timeout` = 30s
- `send_timeout` = 300s

## Root Cause Analysis

The crate's `Inserter` uses **chunked transfer encoding** with an internal spawned tokio task that streams rows to the server. Between chunks, idle gaps form on the connection. When a gap exceeds the server's `http_send_timeout` (30s), the server closes the socket.

Key observations:
1. `with_setting("send_timeout", "600")` has no effect — `http_send_timeout` is enforced at the socket layer and cannot be overridden by client-side settings.
2. `with_setting("max_insert_block_size", "800")` with periodic `end()` helps smaller batches but still fails on ~3000 rows.
3. SSE dataset (~3000+ rows) times out consistently. SZE dataset (~3077 rows) sometimes succeeds, suggesting it's a timing-dependent race.
4. Plain HTTP POST with the same data (full body, single request) completes in 0.1s — the server can accept the data instantly when sent without chunking.

## Expected Behavior

Inserting 3000 rows with 155 columns (~3 MB uncompressed TSV) should succeed without socket timeouts, as the ClickHouse server accepts the same data instantly via direct HTTP POST.

## Workaround

Use `reqwest` with plain HTTP POST:

```rust
let query = "INSERT INTO cn_stock_order FORMAT TabSeparated";
let url = format!("http://host:8123/?user=default&query={}", urlencoding(query));
reqwest::Client::new()
.post(&url)
.body(tsv_body)
.send()
.await?;
```

## Environment
- `clickhouse` crate: 0.15.0 (features: `chrono`, `inserter`, `zstd`)
- ClickHouse server: 24.x on 172.16.210.220:8123
- Rust: edition 2024, tokio 1.x (full features)
- No nginx/reverse proxy (direct ClickHouse connection confirmed)

Contributor guide

Open the contributing guide

Research direction

Start at the Inserter API and its internal spawned task that uses chunked transfer encoding, then reproduce the 3000-row, 155-column case against ClickHouse 24.x. Done means the bulk insert completes without Network("channel closed") or a socket timeout; compare it with the plain reqwest POST behavior described in the report.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, rust
Domain
databases, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.