grafana / grafana/clickhouse-datasource

Clickhouse datasource does not respect Grafana's cookie or OAuth forwarding

Open
#2,139 2 comments 0 reactions 0 assignees View on GitHub
datasource/ClickHouse type/bug
Dominant language
TypeScript
Stars
221
Forks
129
Avg merge
4d 17h
Merged PRs (30d)
21

Description

**What happened**:

When using the Clickhouse HTTP protocol the Clickhouse datasource plugin plugin does not forward HTTP headers and cookies that Grafana is configured to forward.

Specifically:

- `Cookie` entries listed under the data source's `keepCookies` option are not forwarded to ClickHouse; and
- `Authorization` and `X-Id-Token` are not sent to ClickHouse when `oauthPassThru: true` is set on the data source.

Setting `forwardGrafanaHeaders: true` on the data source does not fix this on its own, either alone or in combination with the two options above.

The Grafana Prometheus data source, given the equivalent `keepCookies` / `oauthPassThru` configuration, does forward these headers.

This breaks reverse-proxy setups that place an authorising HTTP proxy in front of ClickHouse and require a per-user cookie or bearer token on every request. The plugin's first outbound call to ClickHouse is an anonymous `queryHello` probe with none of those headers on it, so the proxy rejects it with 403 and the data source fails to connect.

**What you expected to happen**:

With `keepCookies: []` set on the data source and the client request to Grafana carrying that cookie, the outbound HTTP request from the plugin to ClickHouse should include the cookie.

Same for `Authorization` and `X-Id-Token` when `oauthPassThru: true` is set.

[Due to an underlying Grafana limitation](https://github.com/grafana/grafana/issues/87364) general-purpose HTTP header forwarding cannot currently work anyway - but Grafana's cookie-forwarding and its OAuth `authorization_token` forwarding _should_ both work with ClickHouse when suitably enabled.

**How to reproduce it (as minimally and precisely as possible)**:

See attached self-contained repro test case: [repro-2139-v2.zip](https://github.com/user-attachments/files/31622127/repro-2139-v2.zip). It has an embedded README. Test case is LLM-assisted. TL;DR

```sh
unzip repro-2139-v2.zip
cd repro-2139-v2
docker compose up --build --abort-on-container-exit --exit-code-from verifier
```

then check to see log

```
repro-2139-verifier-1 | BUG A REPRODUCED: Cookie x_authorized_project_ids was NOT forwarded to the CH backend under forwardGrafanaHeaders + keepCookies (oauthPassThru=false). See pkg/plugin/driver.go Connect() -- the grafana-header-grafting path is gated on oauthPassThru only.
```

Alternately, explicit manual steps would be too long to list, but here's the general idea:

1. Put an HTTP proxy in front of ClickHouse that returns 403 on any request without `Cookie: my_auth_cookie=...`.
2. Configure a ClickHouse data source pointing at that proxy:

```yaml
type: grafana-clickhouse-datasource
jsonData:
protocol: http
host:
port: 8123
username: grafana_ds
keepCookies:
- my_auth_cookie
forwardGrafanaHeaders: true
```

3. Log in to Grafana with a browser session that carries a valid
`my_auth_cookie`.
4. Run any query against the data source.

Observed: every `/api/ds/query` request fails with `failed to query server hello: [HTTP 403] "missing or invalid cookie"`. The proxy logs an inbound request from the plugin with no `Cookie` header on it.

The when using a matching setup with the Prometheus data source, cookie forwarding works.

**Screenshots**

N/A

**Anything else we need to know?**:

Suspected cause: the plugin's `Connect` runs `db.PingContext` on every call, including the anonymous bootstrap call sqlds makes at data source instantiation with a nil per-query message. That probe carries no user context, so no cookie can attach to it, and the authorising proxy rejects it. `forwardGrafanaHeaders` and `oauthPassThru` only affect per-query dispatch, not this bootstrap call.

Use of a cookie to carry an opaque bearer token is an interim workaround for grafana/grafana#87364 where Grafana itself does not support an allow-list of additional headers to forward from inbound HTTP through to datasources for propagation. Grafana sanitizes them before the datasource plugin can see them on the request, so it's not currently possible for datasources to forward arbitrary headers, only the limited subset that Grafana injects (`Cookie` if cookie allow-list non-empty, `Authorization` and `X-Id-Token` when `oauthPassThru: true`, `X-Grafana-Id`, `X-Grafana-User` if `forwardGrafanaHeaders: true`, and Grafana tracing/dashboard headers). As Grafana _does_ support allow-lists for cookie forwarding, a cookie injected by a reverse-proxy in front of grafana can be used to propagate state through datasources.

Fixing this issue in the plugin will be required to allow the clickhouse plugin to properly support forwarding of client HTTP headers when Grafana upstream gains that ability in a fix to [grafana/grafana#87364](https://github.com/grafana/grafana/issues/87364). With both the plugin fix and a fix to the upstream issue, header forwarding will then work correctly.

It's also important to fix bearer token forwarding per this PR. That allows proxies between Grafana and Clickhouse to consume user identity, group/role membership and embedded custom claims in the OAuth `authorization_token` forwarded by Grafana when `oauthPassThru: true`. The proxy can then use that to modify the Clickhouse request to (e.g.) enforce row-security constraints or specific user/role bindings. (Unfortunately Clickhouse's native JWT auth validation does not support deriving settings from a token, but a proxy in front can do it). The clickhouse plugin does have JWT-forwarding support but it's focused on Clickhouse as a JWT consumer and doesn't match the behaviour of other Grafana plugins' OAuth forwarding.

**Environment**:

- Grafana version: 11.2.0
- Plugin version: 4.20.0
- OS Grafana is installed on: Alpine (Docker), Ubuntu 26.04 host
- User OS & Browser: N/A
- Others:

Contributor guide

Open the contributing guide

Research direction

Start with pkg/plugin/driver.go Connect() and the supplied repro test case; run its docker compose command to observe the missing headers. Compare the ClickHouse forwarding path with the equivalent Prometheus behavior. Done means configured cookies and OAuth authorization headers reach the appropriate ClickHouse requests without breaking the queryHello connection flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, go, grafana
Domain
authentication, backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.