ClickHouse / ClickHouse/clickhouse-cs
ExecuteRawResultAsync / ClickHouseRawResult does not detect in-band mid-stream server exceptions
- Dominant language
- C#
- Stars
- 94
- Forks
- 22
- Avg merge
- 11h 26m
- Merged PRs (30d)
- 22
Description
## Description
When a query fails **after** the ClickHouse HTTP interface has already committed `200 OK` and started streaming rows (e.g. a runtime `throwIf` partway through a large result), the server appends an in-band exception block to the response body and closes the connection.
The native read path handles this: `ClickHouseDataReader.FromHttpResponseAsync` (`ClickHouse.Driver/ADO/Readers/ClickHouseDataReader.cs:66-90`) reads the `X-ClickHouse-Exception-Tag` response header and wraps the body in `ExceptionTagAwareStream`, so `ExecuteReader` raises a clean `ClickHouseServerException` (see `MidStreamExceptionTests.ShouldDetectMidStreamException`).
The **raw / custom-FORMAT streaming path does not**. `ClickHouseClient.ExecuteRawResultAsync` (`ClickHouse.Driver/ClickHouseClient.cs:428-434`) and `ClickHouseCommand.ExecuteRawResultAsync` (`ClickHouse.Driver/ADO/ClickHouseCommand.cs:143-150`) construct `ClickHouseRawResult` directly from the `HttpResponseMessage`, and `ClickHouseRawResult` (`ClickHouse.Driver/ADO/Readers/ClickHouseRawResult.cs:54-72`) hands `response.Content` straight to the caller via `ReadAsStreamAsync` / `ReadAsByteArrayAsync` / `ReadAsStringAsync` / `CopyToAsync`. The exception tag header is never consulted.
Consequences for a consumer streaming `FORMAT CSV` / `JSONEachRow` / `Arrow` / `Parquet` via `ExecuteRawResultAsync`:
- The consumer sees an `System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)` (or a truncated body for the buffered accessors) instead of the server's error.
- When the server-side setting `http_write_exception_in_output_format=1` is enabled, the raw `__exception__ ... __exception__` block is injected verbatim into the caller's data stream — i.e. the caller's CSV/Arrow/Parquet parser is fed garbage bytes — and the stream *still* ends with `HttpIOException`.
- `ReadAsStringAsync` / `ReadAsByteArrayAsync` / `CopyToAsync` are affected the same way; the body is either truncated or contains the exception block, with no `ClickHouseServerException` raised.
This is the .NET analogue of clickhouse-connect#913 (Arrow streaming methods bypassing the in-band exception check).
Related but distinct: #333 covers the same `ExceptionTagAwareStream`-is-only-wired-into-`ClickHouseDataReader` root cause for `ExecuteNonQueryAsync`. This issue is about the raw/custom-FORMAT streaming surface.
## ClickHouse server version
`26.7.1.1315` (local server, HTTP interface). Verified against a running server, not code analysis only.
## Reproduction
Scratch NUnit test added to `ClickHouse.Driver.Tests/ADO/`:
```csharp
using System;
using System.Threading;
using System.Threading.Tasks;
using ClickHouse.Driver.ADO;
using NUnit.Framework;
namespace ClickHouse.Driver.Tests.ADO;
public class RawMidStreamTests : AbstractConnectionTestFixture
{
private const string Query = @"
SELECT toInt32(number) AS n, throwIf(number = 5000000, 'boom') AS e
FROM system.numbers LIMIT 100000000 FORMAT CSV";
[Test]
public async Task RawResultStream_ShouldSurfaceMidStreamException()
{
using var command = connection.CreateCommand();
command.CustomSettings["http_write_exception_in_output_format"] = 1;
command.CommandText = Query;
using var result = await command.ExecuteRawResultAsync(CancellationToken.None);
using var stream = await result.ReadAsStreamAsync();
var tail = string.Empty;
long total = 0;
Exception thrown = null;
var buf = new byte[64 * 1024];
try
{
int n;
while ((n = await stream.ReadAsync(buf, 0, buf.Length)) > 0)
{
total += n;
tail = System.Text.Encoding.UTF8.GetString(buf, 0, n);
}
}
catch (Exception e)
{
thrown = e;
}
TestContext.WriteLine($"BYTES: {total}");
TestContext.WriteLine($"THROWN: {thrown?.GetType().FullName ?? \"\"}");
TestContext.WriteLine($"MSG: {thrown?.Message}");
TestContext.WriteLine($"BODY CONTAINS __exception__: {tail.Contains(\"__exception__\")}");
Assert.That(thrown, Is.InstanceOf(),
"expected clean server exception carrying 'boom'");
}
}
```
**Expected:** a `ClickHouseServerException` carrying `Code: 395 ... boom`, matching what `command.ExecuteReader()` raises for the same query.
**Actual** (`dotnet test -f net10.0 --filter FullyQualifiedName~RawMidStream`):
```
BYTES: 45983730
THROWN: System.Net.Http.HttpIOException
MSG: The response ended prematurely. (ResponseEnded)
BODY CONTAINS __exception__: True
TAIL: boom: while executing 'FUNCTION throwIf(equals(__table1.number, 5000000_UInt32) :: 4, 'boom'_String :: 2)
-> throwIf(equals(__table1.number, 5000000_UInt32), 'boom'_String) UInt8 : 0'.
(FUNCTION_THROW_IF_VALUE_IS_NON_ZERO) (version 26.7.1.1315 (official build))|288 qurkqsppcevjzlmg
Failed RawResultStream_ShouldSurfaceMidStreamException
Expected: instance of
But was:
```
The same test without the `http_write_exception_in_output_format` custom setting also fails with `HttpIOException: The response ended prematurely`.
Note that if the error is raised *before* the server flushes any output (e.g. `throwIf(number = 100000)` with a small result), the response is a non-2xx and `HandleError` already produces a correct `ClickHouseServerException` — the bug only shows once streaming has begun.
## Suggested fix
Plumb the same exception-tag handling used by `ClickHouseDataReader` into the raw path:
- Capture `X-ClickHouse-Exception-Tag` from the `HttpResponseMessage` in `ClickHouseRawResult`'s constructor (`ClickHouse.Driver/ADO/Readers/ClickHouseRawResult.cs:20-23`).
- When the tag is present, wrap the content stream in `ExceptionTagAwareStream` in `ReadAsStreamAsync` and in the buffered accessors (`ReadAsByteArrayAsync`, `ReadAsStringAsync`, `CopyToAsync`), so the `__exception__` block is stripped from the caller's data and re-thrown as `ClickHouseServerException`.
- Consider also surfacing a clean error when the tag is absent but the body ends prematurely, so callers get a ClickHouse-flavoured exception rather than a bare `HttpIOException`.
## Link
Upstream report: https://github.com/ClickHouse/clickhouse-connect/issues/913
Tracking: https://github.com/ClickHouse/integrations-ai-playground/issues/330
Contributor guide
Research direction
Start with ClickHouseDataReader.FromHttpResponseAsync in ClickHouse.Driver/ADO/Readers/ClickHouseDataReader.cs and compare its exception-tag handling with ClickHouseRawResult.cs and the ExecuteRawResultAsync entry points in ClickHouseClient.cs and ClickHouseCommand.cs. Run the RawMidStreamTests reproduction in ClickHouse.Driver.Tests/ADO/ and verify that streaming and buffered raw accessors surface ClickHouseServerException instead of exposing the in-band block or HttpIOException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100