[FEAT - evaluation/losses/plotting] Integrate forecast stability evaluation in utilsforecast

Description

Feature Request: Forecast Stability Evaluation in utilsforecast

Motivation

utilsforecast provides a rich collection of metrics and utilities for evaluating the accuracy of point and probabilistic forecasts. However, in many real-world forecasting applications, accuracy alone is not sufficient to assess forecast quality.

Forecasts are typically generated repeatedly over time and used to support operational decisions such as inventory management, production planning, workforce scheduling, energy allocation, and financial budgeting. In these contexts, forecast stability becomes an important complementary property. A forecasting model may achieve excellent accuracy while producing substantial forecast revisions whenever new observations become available. Such instability can increase operational costs, reduce trust in the forecasting system, and complicate decision-making processes.

Proposed Feature

Introduce support for forecast stability evaluation within utilsforecast.

The implementation could consist of three complementary components:

1. Stability Evaluation Framework

Provide utilities to evaluate forecast stability across multiple forecast origins.

Unlike traditional accuracy metrics, stability metrics require forecasts generated at different points in time and quantify how much forecasts change as new observations become available.

Potential API ideas:

evaluate_stability(
    forecasts,
    metrics=[sMAPC, MASC]
)

or

stability_evaluate(
    df,
    id_col='unique_id',
    time_col='ds',
    model_col='model',
    forecast_origin_col='cutoff'
)

The framework could support:

• Point forecasts
• Quantile forecasts
• Multiple forecast origins
• Aggregation across horizons and series
• Integration with existing utilsforecast.evaluation workflows
• Computation of both horizon-wise and aggregate stability measures

2. Point and Quantile Stability Metrics

Implement stability metrics based on forecast revisions between consecutive forecast origins.

For point forecasts, candidate metrics include:

Metric	Description
MAC	Mean Absolute Change
MAPC	Mean Absolute Percentage Change
sMAPC	Symmetric Mean Absolute Percentage Change
MASC	Mean Absolute Scaled Change
RMSC	Root Mean Squared Change
RMSSC	Root Mean Squared Scaled Change

These metrics quantify the magnitude of forecast updates over time and can be interpreted similarly to traditional forecast accuracy metrics, with forecast errors replaced by forecast revisions.

The same concepts can be naturally extended to quantile forecasts, allowing users to evaluate the temporal consistency of prediction intervals and quantile predictions without requiring full predictive densities.

Examples:

Point Stability Metric	Probabilistic (Quantile) Stability Metric
MAPC	MQPC (Multi-Quantile Percentage Change)
sMAPC	sMQPC (Symmetric Multi-Quantile Percentage Change)
MASC	MQSC (Multi-Quantile Scaled Change)
RMSC	RMQPC (Root Multi-Quantile Squared Change)
RMSSC	RMQSSC (Root Multi-Quantile Squared Scaled Change)

Advantages of this approach include:

• Compatible with quantile forecasting workflows.
• Independent of distributional assumptions.
• Computationally efficient.
• Easy to interpret.
• Applicable to both direct quantile models and conformal prediction intervals.
• Consistent with existing forecast evaluation paradigms.

3. Stability Visualization Utilities

Introduce plotting utilities specifically designed for forecast stability analysis.

Potential visualizations include:

• Stability profiles across forecast horizons.
• Forecast revision trajectories over successive forecast origins.
• Accuracy-versus-stability trade-off plots.

These visualizations would make it easier to diagnose forecast instability and communicate the operational implications of different forecasting strategies.

I would be happy to contribute to the discussion if there is interest in incorporating this functionality into the library.

Use case

Why This Matters

Forecast evaluation has historically focused almost exclusively on predictive accuracy. However, forecasting systems are ultimately decision-support tools, and forecast revisions often carry real operational costs.

A model that is marginally more accurate but substantially less stable may not be preferable in practice. Conversely, some forecasting approaches may achieve similar levels of accuracy while producing significantly more stable forecasts, resulting in lower operational costs and increased user trust.

Evaluating stability alongside accuracy can help practitioners better understand the trade-offs associated with:

• Model selection.
• Retraining frequency.
• Forecast updating policies.
• Ensemble strategies.
• Global versus local forecasting approaches.
• Point versus probabilistic forecasting methodologies.

This is particularly relevant for large-scale forecasting systems where forecasts are continuously updated and directly influence business decisions.

Adding stability evaluation to utilsforecast would:

• Complement the existing accuracy metrics.
• Encourage more comprehensive forecast evaluation.
• Support research on forecast revisions, retraining frequency, and temporal robustness.
• Provide practitioners with tools that better reflect operational requirements.
• Enable systematic comparisons between forecasting models beyond predictive accuracy alone.
• Establish utilsforecast as the first forecasting evaluation library to natively support forecast stability analysis.

The request targets the evaluation/losses/plotting area but does not name files, tests, or a concrete entry point. First clarify the API, metric definitions, supported forecast formats, and which components are in scope with maintainers. Done should include agreed stability evaluation behavior, tests, and plotting support if that remains part of the feature.