Time-series forecasting audit: inference-path gaps and example-correctness defects (follow-up to #1229, #1225, #1161, #1172–#1175)
- Dominant language
- Jupyter Notebook
- Stars
- 4.4k
- Forks
- 565
- Avg merge
- 5d 8m
- Merged PRs (30d)
- 17
Description
## Background
The `ts_forecast` task has a cluster of stale open issues from 2023 (#1229, #1225, #1161, #1172, #1173, #1174, #1175). Before proposing work, I re-verified each one against current `main` (through `38457fba`) with runnable reproductions, in the same spirit as the estimator reproducibility audit (#1540). Results below — several are already fixed and can be closed, several are confirmed live.
## Track A — code
### A1. Timestamp-in-index fails at fit with a misleading error (live, reproducible)
FLAML TS expects the time to be a **column** (`time_col`, defaulting to the first column). When users instead supply the timestamp as the DataFrame **index** — the idiomatic pandas shape, and exactly what the reporter of #1229 did — `fit()` silently takes the *label* column as the time column and raises:
```
ValueError: For '(...ts_forecast...)' task, time column y must contain timestamp values.
```
Repro on current main:
```python
import numpy as np, pandas as pd
from flaml import AutoML
n, horizon = 120, 12
train_df = pd.DataFrame(
{"y": np.sin(np.arange(n) / 6) * 10 + 50},
index=pd.date_range("2018-01-01", periods=n, freq="MS"),
)
automl = AutoML()
automl.fit(dataframe=train_df, label="y", period=horizon, task="ts_forecast")
# ValueError: time column y must contain timestamp values
```
The message points the user at the wrong column and doesn't mention the index. The 2024 comment thread on #1229 confirms real users hit this and had to discover `reset_index()` by trial and error.
**Proposed fix:** in the ts_forecast data-validation path, detect a `DatetimeIndex` (or a RangeIndex alongside a missing/failed time column) and either (a) automatically `reset_index()` it into the time column with an informative log line, or (b) raise a targeted error: *"time appears to be in the DataFrame index; move it into a column (`df.reset_index()`) and pass `time_col=...`"*. I lean (a) for the DatetimeIndex case and (b) for anything ambiguous.
### A2. #1229 / #1225 predict-path failures — apparently fixed; propose regression tests + closure
The originally-reported failures (`KeyError`/`ValueError` at `predict()` with Fourier-feature names like `index_sin1`, `index_hour_cos` on prophet/arima/sarimax; docs-tutorial `KeyError` on 2.1.0) **do not reproduce on current main**. I ran: univariate monthly (lgbm/arima/sarimax), multivariate + exogenous (lgbm/arima/sarimax), and sub-daily hourly (lgbm/sarimax) — all fit+predict cleanly. The intervening TS refactor appears to have fixed the feature-replay-at-predict problem.
**Proposed:** add the three reproductions above as regression tests under `test/`, then close #1229 and #1225 referencing the tests (prophet variant to be covered in CI where prophet is installed).
### A3. #1161 — TFT/panel path breaks on modern pytorch-lightning (live, environment)
`Trainer.__init__() got an unexpected keyword argument 'gpus'` on modern pytorch-lightning; a maintainer (int-chaos) acknowledged in 2024 that the pinned combination is `pytorch-forecasting==0.9.0` + `pytorch-lightning==1.5.10`. Options: pin the `[ts_forecast]` extra accordingly, or update the TFT wrapper to the current lightning API. The former is a small `pyproject.toml` change; the latter is real work I'd scope separately after maintainer input.
## Track B — documentation / examples (all four confirmed in current files)
| # | Defect | Location | Verified |
|---|---|---|---|
| #1172 | **Data leakage** in panel-forecasting example: `avg_volume_by_sku` is a groupby-transform over the *full* dataset before the train/test split | `website/docs/Examples/AutoML-Time series forecast.md` (lines ~730–735) | ✅ present |
| #1175 | **Data leakage** in Forecasting Discrete Values: `above_mean_sales` thresholded on the full-data mean before the split | `notebook/automl_time_series_forecast.ipynb` | ✅ present |
| #1173 | `NameError`: multivariate example builds `multi_df` but calls `automl.fit(dataframe=df, ...)` | same doc, line ~534 | ✅ present |
| #1174 | Comment mismatch: `del multi_df["month"] # remove temperature column…` | same doc, line ~505 | ✅ present |
The two leakage items matter beyond cosmetics — official examples teach the evaluation pattern users copy into production. The fix is to compute derived features within-fold / on train only (or state the simplification explicitly).
## Proposed plan
Following the per-item-PR pattern from #1540, in this order:
1. **PR 1 (Track B):** fix all four doc/notebook defects in one docs PR (they're one file each, same subject) — closes #1172, #1173, #1174, #1175.
2. **PR 2 (A2):** regression tests for the fixed predict paths — closes #1229, #1225.
3. **PR 3 (A1):** DatetimeIndex handling + targeted error message, with tests.
4. **A3:** decide pin-vs-migrate after your input; separate PR either way.
## Questions for maintainers
1. Any objection to closing #1229/#1225 via regression tests given they no longer reproduce?
2. For A1, do you prefer auto-`reset_index()` on DatetimeIndex (convenient, slightly magical) or a targeted error (explicit)?
3. For A3, is pinning the `[ts_forecast]` extra acceptable short-term, or is a lightning-API migration the only acceptable fix?
Happy to adjust scope; will start with PR 1 once this direction is confirmed.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with website/docs/Examples/AutoML-Time series forecast.md and notebook/automl_time_series_forecast.ipynb, then run the supplied ts_forecast reproductions on current main. Review the ts_forecast validation and prediction paths plus the [ts_forecast] dependency configuration; done means the four example defects are corrected, regression coverage exists under test/, and the index and Lightning issues have an agreed, tested resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, pandas, python
- Domain
- data, documentation, machine-learning, testing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100