QuantConnect / QuantConnect/Lean

Post-backtest robustness sections for Report (Monte Carlo, parameter sensitivity, regime classification)

Open
#9,616 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
21.7k
Forks
5.3k
Avg merge
2d 22h
Merged PRs (30d)
34

Description

Expected Behavior

Feature request, not a bug — mapping to the template below.

Lean's Report should offer built-in robustness checks that answer "how much can I trust this backtest result?" Specifically, three opt-in post-backtest sections:

  1. Monte Carlo — resample the trade sequence to produce confidence bands (5th/50th/95th percentile) for CAGR, max drawdown, and Sharpe.
  2. Parameter sensitivity — re-run the backtest with perturbed parameters (±5/10/20%) and report the sensitivity envelope.
  3. Dynamic regime classification — partition returns into bull/bear/sideways/high-vol/low-vol regimes and report per-regime performance.
Actual Behavior

Report currently provides descriptive statistics only (Sharpe, Sortino, drawdown, PSR, hardcoded crisis events, rolling metrics). There is no built-in way to test whether a result is robust to trade ordering, parameter perturbation, or market regime.

Potential Solution

Monte Carlo and regime classification run purely post-backtest on the existing trade log and equity curve — no engine changes, opt-in via config, new sections in Report/template.html. Parameter sensitivity is heavier (it re-runs backtests) and could reuse the existing optimization infrastructure (OptimizationParameter, ParameterSet) or a lighter standalone wrapper.

Asking for design direction before I start: would you accept these, and as one PR or landed separately? My suggestion is to ship Monte Carlo first as a standalone, lowest-risk PR, then discuss the other two.

I'll implement with full unit tests and a regression algorithm.

Prior art: quantstrat (R) has trade-return Monte Carlo; Backtrader has had community MC requests for years with no implementation; Zipline and Freqtrade have none. vectorbt offers resampling and parameter sweeps, so it isn't unprecedented in Python — but it isn't standard.

Checklist
  • I have completely filled out this template
  • I have confirmed that this issue exists on the current master branch
  • I have confirmed that this is not a duplicate issue by searching issues

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Report and Report/template.html, then inspect the existing optimization infrastructure around OptimizationParameter and ParameterSet. Clarify whether Monte Carlo, parameter sensitivity, and regime classification should be separate opt-in sections or a staged change. Done means the agreed sections are implemented with opt-in behavior, full unit tests, and a regression algorithm as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
analytics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.