apache / apache/superset

[SIP-213] Anomaly Detection for Timeseries Charts

Open
#40,520 2 comments 0 reactions 0 assignees View on GitHub
enhancement:request explore:advanced-analysis sip viz:charts:timeseries
Dominant language
Python
Stars
74.8k
Forks
18.3k
Avg merge
2d 5h
Merged PRs (30d)
685

Description

*Please make sure you are familiar with the SIP process documented*
[here](https://github.com/apache/superset/issues/5602). The SIP will be numbered by a committer upon acceptance.

## [SIP-213] Proposal for Anomaly Detection for Timeseries Charts

### Motivation

Superset has forecasting but no way to visually flag anomalies. Users must export data to detect outliers externally.

On management dashboards, anomalies need to be **obvious at a glance** - a revenue drop or traffic spike should stand out immediately without manual inspection. This feature adds that: red scatter points overlaid on timeseries charts, highlighting statistical outliers directly where users are already looking.

It's a purely visual feature (not for alerts/reports), reducing time-to-insight on any dashboard with timeseries data.

### Proposed Change

Add anomaly detection as a post-processing operation for 5 of the ECharts Timeseries chart types (Line, Bar, Area, Smooth Line, Step). Scatter is not yet covered - its control panel doesn't wire in the anomaly detection controls the way the others do. The implementation follows the **exact same architectural pattern** as the existing Prophet forecasting feature:

1. **Control panel section** - "Anomaly Detection" section with enable checkbox and method-specific parameters, placed after the existing Forecast section
2. **Frontend operator** - Generates an `anomaly_detection` entry in the `post_processing` array
3. **Backend post-processing function** - Receives the DataFrame after SQL execution (and after Prophet if enabled), computes anomalies, appends `{column}__anomaly` columns
4. **Frontend rendering** - Recognizes `__anomaly` suffix columns (same pattern as `__yhat`, `__yhat_lower`, `__yhat_upper`) and renders them as red scatter points

#### Detection Methods

| Method | Algorithm | Best for |
| ----------- | ---------------------------------------------------------------- | ----------------------------------------- |
| **Z-Score** | Rolling mean/std deviation, flags points where \|z\| > threshold | General purpose, fast |
| **MAD** | Rolling Median Absolute Deviation. | Data with existing outliers (more robust) |
| **Prophet** | Fits Prophet model, flags points outside confidence interval | Seasonal data with trends |

#### Forecast Integration

When both forecast and anomaly detection are enabled, anomaly detection **automatically runs on the forecast prediction line** (`__yhat` columns). This happens without user configuration because:

- Prophet's post-processing extends the DataFrame with future periods, introducing `NaN` in original columns for those dates
- Anomaly detection skips columns with `NaN` values and skips confidence bounds (`__yhat_lower`, `__yhat_upper`)
- The `__yhat` column has complete data for all dates and is processed automatically

#### Visual Rendering

- **Red scatter points**, using the chart theme's error color (falls back to the series color scale if no theme is available) so the markers stay legible in both light and dark mode
- Minimum symbol size of 10px for clear visibility
- **Excluded from legend** to avoid clutter
- **Tooltip indicator** - `⚠ anomaly` shown when hovering over anomaly points
- Value labels suppressed on anomaly points

### New or Changed Public Interfaces

#### Backend

- **New post-processing operation:** `anomaly_detection` added to `pandas_postprocessing` module
- **Column naming:** Appends `{column}__anomaly` columns to the DataFrame (follows existing `__yhat` convention)

#### Frontend

- **New form data fields** on `EchartsTimeseriesFormData`:
- `anomalyDetectionEnabled` (boolean)
- `anomalyDetectionMethod` (string: `'zscore'` | `'mad'` | `'prophet'`)
- `anomalyDetectionRollingWindow` (number, for zscore/mad)
- `anomalyDetectionSensitivity` (number, for zscore/mad)
- `anomalyDetectionConfidenceInterval` (number, for prophet)
- `anomalyDetectionSeasonalityYearly/Weekly/Daily` (boolean | number | null, for prophet)
- **New enum value:** `Anomaly = '__anomaly'` added to `ForecastSeriesEnum`
- **New type field:** `anomaly?: number` added to `ForecastValue`
- **New TypeScript type:** `PostProcessingAnomalyDetection` added to `PostProcessingRule` union

#### No changes to:

- REST API endpoints
- Database models or configuration
- CLI tools
- Existing saved dashboards/charts (feature is opt-in)

### New dependencies

**None.** Prophet is already an existing optional dependency used by the forecast feature. The Z-Score and MAD methods use only pandas and numpy, which are core dependencies.

### Migration Plan and Compatibility

- **No database migration required** - no new models or schema changes
- **Fully backward compatible** - anomaly detection is disabled by default; existing charts and dashboards are unaffected
- **Saved charts** - existing saved charts will not have anomaly detection fields in their form data, which defaults to disabled (the `ANOMALY_DEFAULT_DATA` provides all defaults)
- **Feature coexistence** - works independently of or alongside the existing forecast feature

### Rejected Alternatives

#### 1. Frontend-only detection (JavaScript)

Rejected because:

- Limited to data visible in the browser (post-pagination/sampling)
- Cannot leverage Prophet for seasonality-aware detection
- Would create inconsistency with the forecast feature which uses backend post-processing

#### 2. Separate "Anomaly Chart" visualization type

Rejected because:

- Anomalies are most useful as an **overlay** on existing timeseries charts
- A separate chart type would force users to duplicate their chart configuration
- The overlay approach matches how Prophet forecast is already rendered

#### 3. Alert/Report integration

Not pursued in this proposal because:

- Anomaly detection's primary value is **visual** - instant recognition on dashboards
- Alert integration would require additional infrastructure (thresholds, notification channels, scheduling)
- Can be added as a follow-up enhancement without changing this implementation

### Samples
Image

Image

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing Prophet forecasting flow and the ECharts Timeseries control panel, frontend rendering, and pandas_postprocessing entry points described in the proposal. Trace how post_processing rules and __yhat columns move from form data through backend processing to chart rendering. Done means the five listed chart types support opt-in z-score, MAD, and Prophet anomalies with red scatter markers, while existing charts remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python, react, typescript
Domain
backend, data-visualization, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.