apache / apache/superset

[SIP-211] Server-side Query Context Generation for Charts

Open
#39,827 7 comments 0 reactions 1 assignee Claimed by @betodealmeida View on GitHub
api design:proposal sip
Dominant language
Python
Stars
74.8k
Forks
18.3k
Avg merge
2d 7h
Merged PRs (30d)
692

Description

### [SIP-211] Server-side Query Context Generation for Charts

#### Motivation

Today, it's impossible to create a chart programmatically via an API call and fetch chart data without first opening the chart in a browser. The reason is because new charts depend on `query_context` being generated from `form_data` by frontend JavaScript (`buildQueryContext` / chart plugin `buildQuery` logic). This creates a gap for backend-only and headless workflows:

- API-created charts may not have `query_context` persisted yet.
- Background jobs (eg alerts/reports) cannot always regenerate `query_context` without running browser JS.
- Integrations that use Superset as a pure backend API have an implicit browser dependency.

We need a backend-native way to generate/re-generate `query_context` from `form_data` without requiring a headless browser. In that the past we have implemented partial solutions to this problem, for example:

- https://github.com/apache/superset/pull/15824
- https://github.com/apache/superset/pull/15846
- https://github.com/apache/superset/pull/15865
- https://github.com/apache/superset/pull/16250
- https://github.com/apache/superset/pull/16303

Note that this would unblock https://github.com/apache/superset/issues/30816.

#### Proposed Change

Introduce an optional internal Node.js service that executes the same JS `buildQueryContext` logic used by Superset frontend plugins, and integrate it with Superset backend as a private dependency.

High-level behavior:

1. **Private sidecar endpoint (not public)**
- A Node.js service exposes an internal endpoint:
- `POST /api/v1/build-query-context` with `{ form_data: ... }`
- It returns generated `query_context` using the same chart plugin buildQuery logic.

2. **Lazy server-side hydration of query_context**
- When Superset backend needs `chart.query_context`:
- If present in DB: use it (current behavior).
- If null/missing: check `QUERY_CONTEXT_SIDECAR_URL`.
- If `QUERY_CONTEXT_SIDECAR_URL` is null/unset: raise an error (same behavior as today).
- If `QUERY_CONTEXT_SIDECAR_URL` is set: call internal Node service, generate `query_context`, persist it in DB, then proceed.
- This enables API-created charts to work without prior browser interaction.

3. **Programmatic refresh support**
- Add (or leverage existing) chart API support for `force=true` semantics:
- regenerate `query_context` from current `form_data` even if one already exists in DB.
- This supports background refresh jobs and consistency after automated edits.

4. **Security / exposure model**
- The Node service is not intended as a public API.
- It should only be reachable by Superset backend over internal network.
- No user-facing/public ingress route should be created for this endpoint in OSS default deployment.
- Even if exposed, the sidecar simply implements logic that is purely functional and open-source.

5. **Operational safeguards**
- Request body size limit.
- Timeout and bounded error logging.
- Optional origin checks / internal allowlist controls.

#### New or Changed Public Interfaces

Potential API/interface changes:

1. **Chart-related API behavior**
- Existing chart data/read workflows may now auto-hydrate missing `query_context` server-side.
- This is a behavior enhancement for API-only workflows.

2. **Optional new parameter**
- Add `force=true` (or equivalent) to relevant chart API(s) to refresh stored `query_context`.
- Exact endpoint(s) to be finalized in implementation discussion.

3. **Configuration**
- New backend config for internal sidecar URL and timeout, eg:
- `QUERY_CONTEXT_SIDECAR_URL`
- request timeout/body limit settings
- `QUERY_CONTEXT_SIDECAR_URL` is optional and defaults to null/unset.
- Behavior with missing `query_context`:
- sidecar URL unset: current error behavior
- sidecar URL set: attempt server-side generation

4. **No new public endpoint in OSS default**
- The Node endpoint is internal only and not documented as a public Superset API.

#### New dependencies

- Node.js runtime service to execute existing frontend query-context logic server-side.
- Reuse existing Superset frontend/plugin JS packages as build-time/runtime dependency of that service.
- No new product-facing browser dependency is introduced.

#### Migration Plan and Compatibility

- No required DB migration expected.
- Backward compatible:
- Existing charts with persisted `query_context` continue to work unchanged.
- Existing API behavior remains valid; this proposal fills missing-query-context cases.
- Rollout strategy:
- No feature flag.
- Optional configuration-only rollout through `QUERY_CONTEXT_SIDECAR_URL`.
- Fallback behavior if sidecar unavailable should be explicit (error vs retry policy).
- Refresh semantics:
- `force=true` should be opt-in and non-breaking.
- Stored dashboards/charts/bookmarks should remain compatible.

#### Rejected Alternatives

1. **Require headless browser in backend jobs**
- Operationally heavy and brittle.
- Hard to scale and maintain.
- Couples backend jobs to frontend runtime environment.

2. **Rewrite JS buildQuery logic in Python**
- Large duplication risk.
- High long-term maintenance burden as chart plugins evolve.

3. **Publicly expose a query-context build endpoint**
- Increases attack surface.
- Not needed for intended OSS behavior; internal-only service is sufficient.

4. **Do nothing (keep browser-first generation only)**
- Continues to block API-first and background-job workflows.
- Keeps implicit frontend dependency in backend use cases.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.