Unclear "Use of Closed Network Connection" Error
- Dominant language
- Go
- Stars
- 428
- Forks
- 84
- Avg merge
- 9d 6h
- Merged PRs (30d)
- 4
Description
### Describe the bug
When using clickhouse-go with clickhouse-cloud, we observe these errors happening during what we suspect to be auto-scaling events on the cloud side.
```
{"time":"2024-05-31T16:22:56.043609257Z","level":"INFO","source":{"function":"github.com/runreveal/lib/await.(*runner).Run.func1","file":"/home/runner/go/pkg/mod/github.com/runreveal/lib/await@v0.0.0-20240514034311-588661551148/await.go","line":110},"msg":"subroutine error: read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!click!house/ch-go@v0.61.5/proto/reader.go:62\n - read tcp 172.30.76.61:54124->35.82.252.60:9440: use of closed network connection"}
```
I believe it's because clickhouse proactively closes network connections that have encountered some kind of exception? It would be nice to have either a clearer error message here, or even better automatic reconnection / retries in the event of a failure here.
Getting this particular error bubbled up to the application isn't particularly useful. The only thing we can do is kill the client and recreate it before re-issuing the query because we can no longer trust the connection.
To be clear: we are not closing the connection on our end, so for whatever reason clickhouse server has to close the connection proactively, I think this library should be able to catch and handle that situation for a cleaner shutdown with a sentinel error value at the very least so we can make a more informed decision about how to handle it.
### Steps to reproduce
1. Create a clickhouse client connection to a clickhouse-cloud instance in Go.
2. Trigger some scaling events while attempting to insert large batches.
3. Observe the error
### Expected behaviour
We get a better error returned or have options to configure retries, or both.
### Error log
```
{"time":"2024-05-31T16:22:56.043609257Z","level":"INFO","source":{"function":"github.com/runreveal/lib/await.(*runner).Run.func1","file":"/home/runner/go/pkg/mod/github.com/runreveal/lib/await@v0.0.0-20240514034311-588661551148/await.go","line":110},"msg":"subroutine error: read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!click!house/ch-go@v0.61.5/proto/reader.go:62\n - read tcp 172.30.76.61:54124->35.82.252.60:9440: use of closed network connection"}
```
### Configuration
#### Environment
* Client version: github.com/ClickHouse/clickhouse-go/v2 v2.23.2
* ch-go version: v0.61.5
* Language version: 1.22.3
* OS: Ubuntu Linux 22.04
#### ClickHouse server
* ClickHouse Server version: 24.2.2
* ClickHouse Server non-default settings, if any: clickhouse-cloud default settings.
* `CREATE TABLE` statements for tables involved:
* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.