apache / apache/superset

A chart's stored query_context is never migrated, so every non-Explore path 500s on charts saved by an older version

Open
#42,926 2 comments 0 reactions 0 assignees View on GitHub
#bug:regression api:charts change:backend
Dominant language
Python
Stars
74.8k
Forks
18.3k
Avg merge
2d 5h
Merged PRs (30d)
685

Description

### Bug description

A chart is persisted twice: `slices.params` holds the chart's settings, and `slices.query_context` holds a query built from those settings. Explore rebuilds the query from `params` at every render and never reads `query_context`. `query_context` is written only when a user saves the chart, and nothing rewrites it on upgrade.

Every consumer that is **not** the browser replays that stored query verbatim:

- `GET /api/v1/chart//data/` — `superset/charts/data/api.py` does `json.loads(chart.query_context)`
- alerts and reports, thumbnails, cache warm-up, CSV/Excel export, which go through the same endpoint
- the MCP server's chart-data tools

So the stored query ages while the query engine moves on, and a chart last saved years ago hands an old query to a current engine. The result is a 500 on all of those paths, while the very same chart renders perfectly in Explore. This makes it look like a data or permission problem rather than a schema-drift problem.

This is the same family as #33152 (*"Viz migrations only migrate the form_data and not the query object. This is problematic as backend processes such as cache warmup or alerts & reports use the saved query object"*) and #31872, but it needs no import and no viz migration to trigger — only time.

#### Symptom 1 — a removed post-processing option

`QueryObject.exec_post_processing` passes the stored options straight through:

```python
df = getattr(pandas_postprocessing, operation)(df, **options)
```

`pivot` used to take `flatten_columns` and `reset_index`. Flattening later became its own `flatten` operation and the parameters were removed, but charts saved before that still carry them in their stored `query_context`. Replaying one gives:

```
TypeError: pivot() got an unexpected keyword argument 'flatten_columns'
```

There is no migration for the stored query, so this is permanent for those charts until somebody opens each one in Explore and re-saves it.

There is also a second defect that hides this from any introspection-based fix. `validate_column_args` in `superset/utils/pandas_postprocessing/utils.py` returns `def wrapped(df, **options)` and does not use `functools.wraps`, so:

```
>>> inspect.signature(pivot)
(df: object, **options: object) -> object
```

Ten operations use that decorator (`aggregate`, `compare`, `contribution`, `cum`, `diff`, `pivot`, `rename`, `rolling`, `select`, `sort`). All of them report a `**kwargs` signature, lose their `__name__` and `__doc__`, and cannot be introspected. `inspect.unwrap` does not help, because without `wraps` there is no `__wrapped__`.

#### Symptom 2 — a time series with no time column

Charts saved by an older frontend store `is_timeseries: true` in the query while the temporal column survives only in `params` as `granularity_sqla`. `superset/models/helpers.py` then raises:

```
Datetime column not provided as part table configuration and is required by this type of chart
```

Again, correct in Explore, 500 everywhere else.

### How to reproduce

1. Take a chart saved by an older Superset (in our instance: an `echarts_timeseries_bar` saved 2023-09 and a `big_number` saved 2023-02). No import, no migration, no edit is needed.
2. Open it in Explore — it renders correctly.
3. Call `GET /api/v1/chart//data/` for the same chart, or attach an alert to it.
4. Symptom 1 or symptom 2 occurs, depending on what the stored query contains.

To confirm the second defect on any version:

```python
import inspect
from superset.utils.pandas_postprocessing import pivot
print(inspect.signature(pivot)) # (df, **options) — the wrapper, not pivot
print(pivot.__name__) # 'wrapped'
```

### Expected results

A stored `query_context` written by an older version stays runnable, or fails with a message that names the cause. A chart that renders in Explore should not 500 on the chart-data endpoint. The post-processing operations should report their own signature and name.

### Actual results

`TypeError: pivot() got an unexpected keyword argument 'flatten_columns'`, or `Datetime column not provided as part table configuration and is required by this type of chart`, on every non-Explore path, permanently, for charts that display correctly in the browser.

### Screenshots/recordings

_No response_

### Superset version

master / latest-dev

### Python version

3.11

### Node version

I don't know

### Browser

Not applicable (server side)

### Additional context

Confirmed on current master (`3b164e4`): `charts/data/api.py` still reads the stored query, `pivot()` still has no `flatten_columns`, the raise in `models/helpers.py` is unchanged, and `validate_column_args` still has no `functools.wraps`. Superset 5.0.0 also has no `flatten_columns`, so symptom 1 is not new.

A forced refresh does not help. `force` only bypasses the result cache; the stored query is the input that builds the SQL.

Measured on a 6.1.0 instance with ~2000 `echarts_timeseries_bar` and ~1300 `big_number` charts: charts saved in 2023 fail, charts saved in 2026 succeed, which matches the age of the stored query rather than the visualisation type.

I have a fix for symptom 1 and for the decorator, and will open a PR that links this issue. Symptom 2 is left out of that PR on purpose: `_apply_granularity` on master has gained its own inference path, so the intended behaviour there is worth a maintainer's opinion before I send code.

### Checklist

- [x] I have searched Superset docs and Slack and didn't find a solution to my problem.
- [x] I have searched the GitHub issue tracker and didn't find a similar bug report.
- [x] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

Contributor guide

Open the contributing guide

Research direction

Start with superset/charts/data/api.py, superset/utils/pandas_postprocessing/utils.py, and superset/models/helpers.py, then reproduce the stored-query failures through the chart-data endpoint. Review the existing post-processing behavior and the older query shapes described here; done means legacy charts no longer fail on non-Explore paths and the affected operations expose their own metadata, with the time-series behavior resolved by maintainer guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
backend-api-design, data-visualization
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.