ClickHouse / ClickHouse/clickhouse-cs

Query parameters sent in URI by default cause HTTP 414 with large IN-lists / vector embeddings

Open
#334 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
94
Forks
22
Avg merge
11h 26m
Merged PRs (30d)
22

Description

## Description

`ClickHouse.Driver` (`ClickHouseClient.PostSqlQueryAsync`) serializes parameterized-query values into the request URI by default (as `param_=...` query-string entries). When parameter values are large — e.g. an `Array(Array(Float32))` of high-dimensional vector embeddings, or a long `IN (...)` list of identifiers — the rendered URI exceeds the URI/header-size limits enforced by common HTTP intermediaries (AWS ALB, nginx `large_client_header_buffers`, CloudFront, etc.) and the request fails with HTTP 414 *Request-URI Too Long* before it ever reaches ClickHouse.

A multipart form-data alternative already exists (`BuildHttpRequestMessageWithFormData`) and is gated by the `ClickHouseClientSettings.UseFormDataParameters` flag. However, this is **opt-in** — `ClickHouseDefaults.UseFormDataParameters = false` (`ClickHouse.Driver/ADO/ClickHouseDefaults.cs:78`), so out-of-the-box every user with non-trivial parameter payloads will hit 414 behind a typical load balancer until they discover and enable the flag.

Code references:

- `ClickHouse.Driver/ADO/ClickHouseDefaults.cs:78` — `public const bool UseFormDataParameters = false;`
- `ClickHouse.Driver/ADO/ClickHouseClientSettings.cs:203` — `public bool UseFormDataParameters { get; init; } = ClickHouseDefaults.UseFormDataParameters;`
- `ClickHouse.Driver/ClickHouseClient.cs:261` — `using var postMessage = Settings.UseFormDataParameters ? BuildHttpRequestMessageWithFormData(...) : BuildHttpRequestMessageWithQueryParams(...);`

This is the same defect class that was reported in `clickhouse-connect` (URL length / HTTP 414 with vector embeddings, source issue ClickHouse/clickhouse-connect#526) and tracked in `clickhouse-java` (#2324). In `clickhouse-java` the opt-in body-mode mechanism shipped but the default was not flipped — the same is true here.

## ClickHouse server version

Code analysis only; not verified against a running server. (Server-side multipart/form-data parameter support has existed for years — see ClickHouse/ClickHouse#8842.)

## Reproduction

Minimal NUnit-style reproduction using this repo's primary API. The buggy default produces a request URI roughly equal to the size of the embedding payload (~1024 floats × ~12 chars ≈ 12 KiB per vector), comfortably exceeding the 8 KiB header limit on a default nginx and most managed proxies.

```csharp
using ClickHouse.Driver;
using ClickHouse.Driver.ADO.Parameters;

// Build a large Array(Array(Float32)) parameter — e.g. 8 × 1024-dim embeddings.
var rows = new float[8][];
for (int i = 0; i < rows.Length; i++)
{
rows[i] = new float[1024];
for (int j = 0; j < 1024; j++) rows[i][j] = (float)(i * 0.001 + j * 0.0001);
}

// Default settings: UseFormDataParameters == false (the bug).
using var client = new ClickHouseClient("Host=localhost");

var parameters = new ClickHouseParameterCollection();
parameters.AddParameter("vecs", rows);

// Behind any proxy with a typical 8 KiB header limit (AWS ALB, default nginx,
// CloudFront, etc.), this request fails with HTTP 414 Request-URI Too Long
// before reaching ClickHouse, because the param_vecs=... entry is appended
// to the URI by BuildHttpRequestMessageWithQueryParams.
using var reader = await client.ExecuteReaderAsync(
"SELECT length(arrayElement({vecs:Array(Array(Float32))}, 1))",
parameters);

// Workaround that should not be required: opt into body-mode parameters.
// var settings = new ClickHouseClientSettings("Host=localhost")
// {
// UseFormDataParameters = true,
// };
// using var fixedClient = new ClickHouseClient(settings);
```

**Expected**: query succeeds regardless of parameter size, as it does when `UseFormDataParameters = true`.

**Actual** (with default settings, behind a typical proxy): HTTP 414 Request-URI Too Long.

An equivalent reproduction without a proxy: log the outgoing URI from `BuildHttpRequestMessageWithQueryParams` and observe that its length grows linearly with parameter size and quickly exceeds 8 KiB for the payload above.

## Suggested fix

Two reasonable options, in order of preference:

1. **Flip the default**: change `ClickHouseDefaults.UseFormDataParameters` to `true`. The multipart body path is already exercised and is strictly safer for large parameter payloads; the URI path remains available for users who explicitly opt out. The server has supported params-in-body for years.
2. **Auto-promote** to the form-data path when the rendered query string would exceed a safe threshold (e.g. 4 KiB), keeping the URI path for small/cheap queries. This avoids any behavior change for users who explicitly set `UseFormDataParameters = false`.

If the default stays `false`, at minimum surface the 414 failure mode in `ClickHouseClientSettings.UseFormDataParameters` XML docs and log a one-time WARN when an outgoing request URI exceeds a threshold.

## Link

- Source bug: ClickHouse/clickhouse-connect#526
- Sibling client tracker: ClickHouse/clickhouse-java#2324 (shipped opt-in, default unchanged)
- Central tracking issue: ClickHouse/integrations-ai-playground#150

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.