bloomberg / bloomberg/causal-ts
regime_discovery.ipynb: detector rankings come from a single seed and don't reproduce
- Dominant language
- Python
- Stars
- 36
- Forks
- 15
- Avg merge
- 4h 1m
- Merged PRs (30d)
- 36
Description
Section 8 of `examples/regime_discovery.ipynb` compares changepoint detectors across three regime-change types, and section 9 turns those results into a "which detector for which change type" recommendation table. Each row rests on **one dataset at one seed**, and the rankings do not survive repetition.
Re-running all three comparisons over 10 seeds, same generative process, F1 mean ± sd:
| Detector | Variance shift | Shape shift | Gradual drift |
|---|---|---|---|
| Ensemble | **1.000 ± 0.000** | 0.927 ± 0.125 | 0.725 ± 0.149 |
| PELT | 0.967 ± 0.105 | **0.986 ± 0.045** | **0.854 ± 0.091** |
| HMM | **0.986 ± 0.045** | **0.986 ± 0.045** | 0.724 ± 0.129 |
| HMM (auto) | **0.986 ± 0.045** | 0.886 ± 0.314 | 0.790 ± 0.306 |
| KS | **0.986 ± 0.045** | 0.907 ± 0.105 | 0.776 ± 0.077 |
| Levene | **0.986 ± 0.045** | 0.870 ± 0.109 | 0.714 ± 0.080 |
| CUSUM (mean) | 0.957 ± 0.069 | **0.986 ± 0.045** | 0.688 ± 0.105 |
| CUSUM (var/both) | 0.942 ± 0.125 | 0.837 ± 0.116 | 0.698 ± 0.125 |
| Baseline | 0.779 ± 0.088 | 0.665 ± 0.083 | 0.654 ± 0.196 |
Against the single-seed numbers currently in the notebook, `Ensemble` moves from near-worst to best on variance shift and `PELT` moves from joint-first to sixth. No detector wins everywhere. `HMM (auto)` is worth a look — sd above 0.30 on two of three, with a minimum of 0.000, and the run emits `Model is not converging`.
**What needs doing**
1. **Make the data reproducible.** Cells 3, 20, 24 and 27 generate data with bare `np.random.randn()`, and the only seeding is a single `np.random.seed(42)` in cell 1. Each dataset therefore depends on cumulative RNG state and changes if cells are run out of order. Give each generator its own `np.random.default_rng(seed)`.
2. **Run each comparison over several seeds** — 10 is enough to show the effect — and report **mean ± sd** rather than a single number.
3. **Handle `hmmlearn` being absent.** `HMMRegimeDetector` requires it and it is deliberately not a dependency of `causalts`. Skip the HMM rows on `ImportError`, with a note that `pip install hmmlearn` enables them, so the notebook runs on a plain install.
4. **Update section 9 so its recommendations follow from the new numbers.** The current table names a single winner per row, which error bars won't support. One correction that's independent of the sweep: PELT's stated limitation "Only mean shifts" is wrong — what PELT detects is set by its cost function, and `PeltRegimeDetector` already has a `signal="volatility"` option.
**Keep HMM.** It scores 0.986 / 0.986 / 0.724 and its worst single seed anywhere is 0.600 — it never collapses. This is about the methodology, not about demoting HMM.
The sweep used 10 seeds and F1 only, with `default_rng(seed)` in place of the notebook's global `np.random` — the same generative process, but not the same draws.
Contributor guide
Research direction
Start with examples/regime_discovery.ipynb, especially cells 1, 3, 20, 24 and 27, then inspect sections 8 and 9. Run the notebook with multiple seeds and verify that results report mean ± sd, HMM rows are skipped with an installation note when hmmlearn is absent, and the recommendations and PELT limitation match the results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, numpy, python
- Domain
- data, machine-learning
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100