Bug Report: Bayesian experiments flag results significant while the credible interval still contains 0

Bug Description

In the Bayesian stats engine, a variant can be flagged significant: true while its 95% credible interval still contains 0. The frequentist engine has no such gap. The two engines use decision rules that aren't comparable, and the Bayesian one is internally inconsistent with the interval it ships alongside it.

This isn't only an API-level flag: the metrics chart draws the violin between the credible-interval bounds but colors it by significant, so a bar visibly straddling the 0% line renders as a green (or red) winner. The "Is my variant significant?" tooltip — shown for both methods — states outright: "When an interval doesn't cross the 0% line, the result is significant." That's the exact invariant the Bayesian path breaks.

Root cause

Both paths assign significant in products/experiments/backend/hogql_queries/utils.py:

• Frequentist — utils.py:555 → result.is_significant, defined at stats/frequentist/tests.py:141 (and :223 for the sequential test) as p_value < self.alpha. The p-value is two-sided and the CI uses t.ppf(1 - alpha/2), so p < 0.05 ⟺ 95% CI excludes 0. Coherent by construction.

• Bayesian — utils.py:659 → result.is_decisive, defined at stats/bayesian/tests.py:65-67:

  @property
  def is_decisive(self) -> bool:
      """Whether result shows clear preference (chance to win > ci_level or < 1 - ci_level)."""
      return self.chance_to_win > self.ci_level or self.chance_to_win < 1 - self.ci_level
  

  chance_to_win is a one-sided posterior tail — norm.sf(0, μ, σ) (bayesian/utils.py:176). But the credible interval it's presented next to is two-sided equal-tailed: alpha = 1 - self.ci_level (tests.py:195) then norm.ppf([alpha/2, 1 - alpha/2]) (bayesian/utils.py:201).

  A one-sided probability is being thresholded at a two-sided level. With ci_level = 0.95:

  • chance_to_win > 0.95 ⟺ z > 1.645
  • credible interval excludes 0 ⟺ z > 1.960

  Every result in between is reported significant with an interval spanning 0.

The gap exists at every configurable level, not just the default:

ci_level	decisive when z >	interval excludes 0 when z >	chance_to_win values in the gap
0.80	0.8416	1.2816	0.800 – 0.900
0.90	1.2816	1.6449	0.900 – 0.950
0.95	1.6449	1.9600	0.950 – 0.975
0.99	2.3263	2.5758	0.990 – 0.995

How to reproduce

Against products/experiments/stats at fb633ba, using the shipped classes directly (control 2000/20000, treatment n=20000, relative difference, default priors):

from products.experiments.stats.bayesian.method import BayesianConfig, BayesianMethod
from products.experiments.stats.frequentist.method import FrequentistConfig, FrequentistMethod
from products.experiments.stats.shared.enums import DifferenceType
from products.experiments.stats.shared.statistics import ProportionStatistic

bayes = BayesianMethod(BayesianConfig(ci_level=0.95, difference_type=DifferenceType.RELATIVE))
freq = FrequentistMethod(FrequentistConfig(alpha=0.05, difference_type=DifferenceType.RELATIVE))

control = ProportionStatistic(n=20000, sum=2000)
for k in (2101, 2104, 2113, 2122, 2125):
    treatment = ProportionStatistic(n=20000, sum=k)
    b, f = bayes.run_test(treatment, control), freq.run_test(treatment, control)
    lo, hi = b.credible_interval
    print(k, round(b.chance_to_win, 4), b.is_decisive, (round(lo, 4), round(hi, 4)),
          lo < 0 < hi, round(f.p_value, 3), f.is_significant)

treat conv     ctw  signif       credible interval    CI has 0 |     p  freq sig
      2101  0.9478   False  [-0.0104, +0.1114]           True | 0.104     False
      2104  0.9526    True  [-0.0090, +0.1130]           True | 0.095     False   <== significant, interval spans 0
      2113  0.9648    True  [-0.0047, +0.1177]           True | 0.070     False   <==
      2122  0.9743    True  [-0.0004, +0.1224]           True | 0.051     False   <==
      2125  0.9769    True  [+0.0011, +0.1239]          False | 0.046      True

Over a 20,000-run random grid of proportion experiments (n from 500–50,000, base rate 2–50%, true lift ~ N(0, 6%)), checking significant against "interval excludes 0":

    Bayesian (is_decisive): 12015 flagged significant, 1210 of them (10.1%) have a credible interval containing 0
       Frequentist (fixed): 10933 flagged significant,    0 incoherent
  Frequentist (sequential):  7153 flagged significant,    0 incoherent

The grid concentrates lift near the decision boundary, so 10% is not a production rate — but the error is strictly one-directional. Across 20k runs there was not a single case of the reverse (interval excluding 0 while not flagged significant).

Suggested fix

Two coherent options; the first is a one-line change:

1. Match the interval — threshold chance_to_win at 1 - (1 - ci_level)/2 (0.975 for a 95% level), making significant exactly equivalent to "credible interval excludes 0", matching the frequentist path and the tooltip's own description.
2. Match the threshold — keep the 0.95 one-sided rule and report a one-sided credible bound instead. Larger UI change, and it would make the Bayesian and frequentist charts mean different things.

Option 1 preserves what the UI already claims. If the 0.95 one-sided rule is the deliberate product decision (it's a defensible Bayesian decision rule on its own — the legacy engine at funnels_statistics_v2.py:140 used win-probability ≥ 0.9 plus an expected-loss cap, and never claimed interval duality), then the tooltip copy and the chart's significance coloring need to stop implying the interval is the decision boundary.

Worth noting there is currently no test coverage for is_decisive — products/experiments/stats/tests/test_bayesian.py never references it. A regression test asserting is_decisive == (0 not in credible_interval) would pin whichever semantics you choose.

Additional context

Possibly related: #62236 (credible interval not reflecting the configured statistics level). That one is about ci_level not reaching the displayed interval; this one is about the threshold semantics, but they touch adjacent code and the fixes should probably be considered together.

Debug info

Found by code inspection against PostHog/posthog @ fb633ba8189ecbdcff6e231c41572807594955cb (master).
Reproduced by running products/experiments/stats classes directly — no PostHog instance required.

Start with is_decisive in products/experiments/stats/bayesian/tests.py and the credible-interval calculation in products/experiments/stats/bayesian/utils.py, then inspect their use in products/experiments/backend/hogql_queries/utils.py. Add regression coverage in products/experiments/stats/tests/test_bayesian.py; done means the chosen significance semantics and the displayed interval are consistent at configurable confidence levels.