QuantConnect / QuantConnect/Lean
Post-backtest robustness sections for Report (Monte Carlo, parameter sensitivity, regime classification)
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 21.7k
- Forks
- 5.3k
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 34
Description
Expected Behavior
Feature request, not a bug — mapping to the template below.
Lean's Report should offer built-in robustness checks that answer "how much can I trust this backtest result?" Specifically, three opt-in post-backtest sections:
- Monte Carlo — resample the trade sequence to produce confidence bands (5th/50th/95th percentile) for CAGR, max drawdown, and Sharpe.
- Parameter sensitivity — re-run the backtest with perturbed parameters (±5/10/20%) and report the sensitivity envelope.
- Dynamic regime classification — partition returns into bull/bear/sideways/high-vol/low-vol regimes and report per-regime performance.
Actual Behavior
Report currently provides descriptive statistics only (Sharpe, Sortino, drawdown, PSR, hardcoded crisis events, rolling metrics). There is no built-in way to test whether a result is robust to trade ordering, parameter perturbation, or market regime.
Potential Solution
Monte Carlo and regime classification run purely post-backtest on the existing trade log and equity curve — no engine changes, opt-in via config, new sections in Report/template.html. Parameter sensitivity is heavier (it re-runs backtests) and could reuse the existing optimization infrastructure (OptimizationParameter, ParameterSet) or a lighter standalone wrapper.
Asking for design direction before I start: would you accept these, and as one PR or landed separately? My suggestion is to ship Monte Carlo first as a standalone, lowest-risk PR, then discuss the other two.
I'll implement with full unit tests and a regression algorithm.
Prior art: quantstrat (R) has trade-return Monte Carlo; Backtrader has had community MC requests for years with no implementation; Zipline and Freqtrade have none. vectorbt offers resampling and parameter sweeps, so it isn't unprecedented in Python — but it isn't standard.
Checklist
- I have completely filled out this template
- I have confirmed that this issue exists on the current
masterbranch - I have confirmed that this is not a duplicate issue by searching issues
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Report and Report/template.html, then inspect the existing optimization infrastructure around OptimizationParameter and ParameterSet. Clarify whether Monte Carlo, parameter sensitivity, and regime classification should be separate opt-in sections or a staged change. Done means the agreed sections are implemented with opt-in behavior, full unit tests, and a regression algorithm as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- analytics
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100