ClickHouse / ClickHouse/clickhouse-cs
Query parameters sent in URI by default cause HTTP 414 with large IN-lists / vector embeddings
- Dominant language
- C#
- Stars
- 94
- Forks
- 22
- Avg merge
- 11h 26m
- Merged PRs (30d)
- 22
Description
## Description
`ClickHouse.Driver` (`ClickHouseClient.PostSqlQueryAsync`) serializes parameterized-query values into the request URI by default (as `param_=...` query-string entries). When parameter values are large — e.g. an `Array(Array(Float32))` of high-dimensional vector embeddings, or a long `IN (...)` list of identifiers — the rendered URI exceeds the URI/header-size limits enforced by common HTTP intermediaries (AWS ALB, nginx `large_client_header_buffers`, CloudFront, etc.) and the request fails with HTTP 414 *Request-URI Too Long* before it ever reaches ClickHouse.
A multipart form-data alternative already exists (`BuildHttpRequestMessageWithFormData`) and is gated by the `ClickHouseClientSettings.UseFormDataParameters` flag. However, this is **opt-in** — `ClickHouseDefaults.UseFormDataParameters = false` (`ClickHouse.Driver/ADO/ClickHouseDefaults.cs:78`), so out-of-the-box every user with non-trivial parameter payloads will hit 414 behind a typical load balancer until they discover and enable the flag.
Code references:
- `ClickHouse.Driver/ADO/ClickHouseDefaults.cs:78` — `public const bool UseFormDataParameters = false;`
- `ClickHouse.Driver/ADO/ClickHouseClientSettings.cs:203` — `public bool UseFormDataParameters { get; init; } = ClickHouseDefaults.UseFormDataParameters;`
- `ClickHouse.Driver/ClickHouseClient.cs:261` — `using var postMessage = Settings.UseFormDataParameters ? BuildHttpRequestMessageWithFormData(...) : BuildHttpRequestMessageWithQueryParams(...);`
This is the same defect class that was reported in `clickhouse-connect` (URL length / HTTP 414 with vector embeddings, source issue ClickHouse/clickhouse-connect#526) and tracked in `clickhouse-java` (#2324). In `clickhouse-java` the opt-in body-mode mechanism shipped but the default was not flipped — the same is true here.
## ClickHouse server version
Code analysis only; not verified against a running server. (Server-side multipart/form-data parameter support has existed for years — see ClickHouse/ClickHouse#8842.)
## Reproduction
Minimal NUnit-style reproduction using this repo's primary API. The buggy default produces a request URI roughly equal to the size of the embedding payload (~1024 floats × ~12 chars ≈ 12 KiB per vector), comfortably exceeding the 8 KiB header limit on a default nginx and most managed proxies.
```csharp
using ClickHouse.Driver;
using ClickHouse.Driver.ADO.Parameters;
// Build a large Array(Array(Float32)) parameter — e.g. 8 × 1024-dim embeddings.
var rows = new float[8][];
for (int i = 0; i < rows.Length; i++)
{
rows[i] = new float[1024];
for (int j = 0; j < 1024; j++) rows[i][j] = (float)(i * 0.001 + j * 0.0001);
}
// Default settings: UseFormDataParameters == false (the bug).
using var client = new ClickHouseClient("Host=localhost");
var parameters = new ClickHouseParameterCollection();
parameters.AddParameter("vecs", rows);
// Behind any proxy with a typical 8 KiB header limit (AWS ALB, default nginx,
// CloudFront, etc.), this request fails with HTTP 414 Request-URI Too Long
// before reaching ClickHouse, because the param_vecs=... entry is appended
// to the URI by BuildHttpRequestMessageWithQueryParams.
using var reader = await client.ExecuteReaderAsync(
"SELECT length(arrayElement({vecs:Array(Array(Float32))}, 1))",
parameters);
// Workaround that should not be required: opt into body-mode parameters.
// var settings = new ClickHouseClientSettings("Host=localhost")
// {
// UseFormDataParameters = true,
// };
// using var fixedClient = new ClickHouseClient(settings);
```
**Expected**: query succeeds regardless of parameter size, as it does when `UseFormDataParameters = true`.
**Actual** (with default settings, behind a typical proxy): HTTP 414 Request-URI Too Long.
An equivalent reproduction without a proxy: log the outgoing URI from `BuildHttpRequestMessageWithQueryParams` and observe that its length grows linearly with parameter size and quickly exceeds 8 KiB for the payload above.
## Suggested fix
Two reasonable options, in order of preference:
1. **Flip the default**: change `ClickHouseDefaults.UseFormDataParameters` to `true`. The multipart body path is already exercised and is strictly safer for large parameter payloads; the URI path remains available for users who explicitly opt out. The server has supported params-in-body for years.
2. **Auto-promote** to the form-data path when the rendered query string would exceed a safe threshold (e.g. 4 KiB), keeping the URI path for small/cheap queries. This avoids any behavior change for users who explicitly set `UseFormDataParameters = false`.
If the default stays `false`, at minimum surface the 414 failure mode in `ClickHouseClientSettings.UseFormDataParameters` XML docs and log a one-time WARN when an outgoing request URI exceeds a threshold.
## Link
- Source bug: ClickHouse/clickhouse-connect#526
- Sibling client tracker: ClickHouse/clickhouse-java#2324 (shipped opt-in, default unchanged)
- Central tracking issue: ClickHouse/integrations-ai-playground#150
Contributor guide
Assessment
This issue has not been assessed yet.