A chart's stored query_context is never migrated, so every non-Explore path 500s on charts saved by an older version
- Dominant language
- Python
- Stars
- 74.8k
- Forks
- 18.3k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 685
Description
### Bug description
A chart is persisted twice: `slices.params` holds the chart's settings, and `slices.query_context` holds a query built from those settings. Explore rebuilds the query from `params` at every render and never reads `query_context`. `query_context` is written only when a user saves the chart, and nothing rewrites it on upgrade.
Every consumer that is **not** the browser replays that stored query verbatim:
- `GET /api/v1/chart//data/` — `superset/charts/data/api.py` does `json.loads(chart.query_context)`
- alerts and reports, thumbnails, cache warm-up, CSV/Excel export, which go through the same endpoint
- the MCP server's chart-data tools
So the stored query ages while the query engine moves on, and a chart last saved years ago hands an old query to a current engine. The result is a 500 on all of those paths, while the very same chart renders perfectly in Explore. This makes it look like a data or permission problem rather than a schema-drift problem.
This is the same family as #33152 (*"Viz migrations only migrate the form_data and not the query object. This is problematic as backend processes such as cache warmup or alerts & reports use the saved query object"*) and #31872, but it needs no import and no viz migration to trigger — only time.
#### Symptom 1 — a removed post-processing option
`QueryObject.exec_post_processing` passes the stored options straight through:
```python
df = getattr(pandas_postprocessing, operation)(df, **options)
```
`pivot` used to take `flatten_columns` and `reset_index`. Flattening later became its own `flatten` operation and the parameters were removed, but charts saved before that still carry them in their stored `query_context`. Replaying one gives:
```
TypeError: pivot() got an unexpected keyword argument 'flatten_columns'
```
There is no migration for the stored query, so this is permanent for those charts until somebody opens each one in Explore and re-saves it.
There is also a second defect that hides this from any introspection-based fix. `validate_column_args` in `superset/utils/pandas_postprocessing/utils.py` returns `def wrapped(df, **options)` and does not use `functools.wraps`, so:
```
>>> inspect.signature(pivot)
(df: object, **options: object) -> object
```
Ten operations use that decorator (`aggregate`, `compare`, `contribution`, `cum`, `diff`, `pivot`, `rename`, `rolling`, `select`, `sort`). All of them report a `**kwargs` signature, lose their `__name__` and `__doc__`, and cannot be introspected. `inspect.unwrap` does not help, because without `wraps` there is no `__wrapped__`.
#### Symptom 2 — a time series with no time column
Charts saved by an older frontend store `is_timeseries: true` in the query while the temporal column survives only in `params` as `granularity_sqla`. `superset/models/helpers.py` then raises:
```
Datetime column not provided as part table configuration and is required by this type of chart
```
Again, correct in Explore, 500 everywhere else.
### How to reproduce
1. Take a chart saved by an older Superset (in our instance: an `echarts_timeseries_bar` saved 2023-09 and a `big_number` saved 2023-02). No import, no migration, no edit is needed.
2. Open it in Explore — it renders correctly.
3. Call `GET /api/v1/chart//data/` for the same chart, or attach an alert to it.
4. Symptom 1 or symptom 2 occurs, depending on what the stored query contains.
To confirm the second defect on any version:
```python
import inspect
from superset.utils.pandas_postprocessing import pivot
print(inspect.signature(pivot)) # (df, **options) — the wrapper, not pivot
print(pivot.__name__) # 'wrapped'
```
### Expected results
A stored `query_context` written by an older version stays runnable, or fails with a message that names the cause. A chart that renders in Explore should not 500 on the chart-data endpoint. The post-processing operations should report their own signature and name.
### Actual results
`TypeError: pivot() got an unexpected keyword argument 'flatten_columns'`, or `Datetime column not provided as part table configuration and is required by this type of chart`, on every non-Explore path, permanently, for charts that display correctly in the browser.
### Screenshots/recordings
_No response_
### Superset version
master / latest-dev
### Python version
3.11
### Node version
I don't know
### Browser
Not applicable (server side)
### Additional context
Confirmed on current master (`3b164e4`): `charts/data/api.py` still reads the stored query, `pivot()` still has no `flatten_columns`, the raise in `models/helpers.py` is unchanged, and `validate_column_args` still has no `functools.wraps`. Superset 5.0.0 also has no `flatten_columns`, so symptom 1 is not new.
A forced refresh does not help. `force` only bypasses the result cache; the stored query is the input that builds the SQL.
Measured on a 6.1.0 instance with ~2000 `echarts_timeseries_bar` and ~1300 `big_number` charts: charts saved in 2023 fail, charts saved in 2026 succeed, which matches the age of the stored query rather than the visualisation type.
I have a fix for symptom 1 and for the decorator, and will open a PR that links this issue. Symptom 2 is left out of that PR on purpose: `_apply_granularity` on master has gained its own inference path, so the intended behaviour there is worth a maintainer's opinion before I send code.
### Checklist
- [x] I have searched Superset docs and Slack and didn't find a solution to my problem.
- [x] I have searched the GitHub issue tracker and didn't find a similar bug report.
- [x] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
Contributor guide
Research direction
Start with superset/charts/data/api.py, superset/utils/pandas_postprocessing/utils.py, and superset/models/helpers.py, then reproduce the stored-query failures through the chart-data endpoint. Review the existing post-processing behavior and the older query shapes described here; done means legacy charts no longer fail on non-Explore paths and the affected operations expose their own metadata, with the time-series behavior resolved by maintainer guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- backend-api-design, data-visualization
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100