ClickHouse / ClickHouse/ch-go

Unclear "Use of Closed Network Connection" Error

Open
#405 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
428
Forks
84
Avg merge
9d 6h
Merged PRs (30d)
4

Description

### Describe the bug

When using clickhouse-go with clickhouse-cloud, we observe these errors happening during what we suspect to be auto-scaling events on the cloud side.

```
{"time":"2024-05-31T16:22:56.043609257Z","level":"INFO","source":{"function":"github.com/runreveal/lib/await.(*runner).Run.func1","file":"/home/runner/go/pkg/mod/github.com/runreveal/lib/await@v0.0.0-20240514034311-588661551148/await.go","line":110},"msg":"subroutine error: read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!click!house/ch-go@v0.61.5/proto/reader.go:62\n - read tcp 172.30.76.61:54124->35.82.252.60:9440: use of closed network connection"}
```

I believe it's because clickhouse proactively closes network connections that have encountered some kind of exception? It would be nice to have either a clearer error message here, or even better automatic reconnection / retries in the event of a failure here.

Getting this particular error bubbled up to the application isn't particularly useful. The only thing we can do is kill the client and recreate it before re-issuing the query because we can no longer trust the connection.

To be clear: we are not closing the connection on our end, so for whatever reason clickhouse server has to close the connection proactively, I think this library should be able to catch and handle that situation for a cleaner shutdown with a sentinel error value at the very least so we can make a more informed decision about how to handle it.

### Steps to reproduce
1. Create a clickhouse client connection to a clickhouse-cloud instance in Go.
2. Trigger some scaling events while attempting to insert large batches.
3. Observe the error

### Expected behaviour

We get a better error returned or have options to configure retries, or both.

### Error log

```
{"time":"2024-05-31T16:22:56.043609257Z","level":"INFO","source":{"function":"github.com/runreveal/lib/await.(*runner).Run.func1","file":"/home/runner/go/pkg/mod/github.com/runreveal/lib/await@v0.0.0-20240514034311-588661551148/await.go","line":110},"msg":"subroutine error: read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!click!house/ch-go@v0.61.5/proto/reader.go:62\n - read tcp 172.30.76.61:54124->35.82.252.60:9440: use of closed network connection"}
```

### Configuration
#### Environment
* Client version: github.com/ClickHouse/clickhouse-go/v2 v2.23.2
* ch-go version: v0.61.5
* Language version: 1.22.3
* OS: Ubuntu Linux 22.04

#### ClickHouse server
* ClickHouse Server version: 24.2.2
* ClickHouse Server non-default settings, if any: clickhouse-cloud default settings.
* `CREATE TABLE` statements for tables involved:
* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.