google / google/meridian

MCMC non-convergence and catastrophic predictive_accuracy on 1.6.0 with official tfp-nightly dependency

Open
#1,624 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.5k
Forks
294
Avg merge
1d 21h
Merged PRs (30d)
47

Description

### Summary

Upgrading from `google-meridian==1.4.0` to `google-meridian==1.6.0` with the **official** dependency stack from [pyproject.toml](https://github.com/google/meridian/blob/v1.6.0/pyproject.toml) (`tfp-nightly==0.26.0.dev20260130`) causes complete MCMC non-convergence and nonsensical goodness-of-fit metrics, while the same training data, config, and priors work on 1.4.0.

### Environment

- Platform: linux aarch64 (Docker on Apple Silicon)
- Python: 3.11
- `google-meridian`: 1.6.0
- `tensorflow`: 2.20.0
- `tfp-nightly`: 0.26.0.dev20260130 (as required by pyproject.toml)
- `numpy`: 2.3.5 (< 2.4)

Working baseline (1.4.0):

- `google-meridian`: 1.4.0
- `tensorflow`: 2.20.0
- `tensorflow-probability`: 0.25.0 (stable; declared in [v1.4.0 pyproject.toml](https://github.com/google/meridian/blob/v1.4.0/pyproject.toml))

### Reproduction

National/geo MMM, `kpi_type=non_revenue`, no `revenue_per_kpi`, ROI priors via `PriorDistribution(roi_m=LogNormal(...))`, holdout_id mask, MCMC: 8 chains / 2000 adapt / 1000 burnin / 2000 keep.

Install official 1.6.0 stack:

```bash
pip install 'numpy<2.4' 'google-meridian==1.6.0'
pip freeze | grep -iE 'meridian|tensorflow|tfp'
```

Train model, then:

```python
from meridian.analysis import analyzer
from meridian.analysis.review import reviewer

analysis = analyzer.Analyzer(mmm)
print(analysis.predictive_accuracy(use_kpi=True).to_dataframe())

health = reviewer.ModelReviewer(mmm).run()
print(health.overall_status, health.health_score, health.summary_message)
```

### Actual results (1.6.0 + official tfp-nightly)

**Predictive accuracy (ALL geo model):**

| metric | geo_granularity | evaluation_set | value |
|--------|-----------------|----------------|-------|
| R_Squared | national | All Data | -68817.18 |
| MAPE | national | All Data | 62.83 |
| wMAPE | national | All Data | 64.02 |

**Health check:**

- `overall_status`: FAIL
- `health_score`: 0.0
- `max_r_hat` for `roi_m`: **338660163584.00**
- Message: "Failed: Model did not converge. Other checks were skipped."

### Expected results (1.4.0 + tensorflow-probability 0.25.0)

Same data and config on 1.4.0:

| metric | geo_granularity | evaluation_set | value |
|--------|-----------------|----------------|-------|
| R_Squared | national | All Data | 0.908 |
| MAPE | national | All Data | 0.051 |
| wMAPE | national | All Data | 0.053 |

### Notes

1. `tfp-nightly` is an **explicit** dependency in 1.5.3+ pyproject.toml, not an accidental resolver artifact.
2. 1.5.3 also declares the same `tfp-nightly` pin; we have not fully validated whether 1.5.3 official stack reproduces the issue.
3. Replacing `tfp-nightly` with `tensorflow-probability==0.25.0` after installing 1.6.0 is a possible workaround but violates declared dependencies and may break if 1.6.0 calls 0.26-only APIs.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the v1.6.0 and v1.4.0 pyproject.toml dependency declarations, then reproduce the reported training setup with the official v1.6.0 stack. Inspect analyzer.Analyzer.predictive_accuracy and reviewer.ModelReviewer outputs alongside MCMC convergence; done means the official dependency configuration no longer produces the reported non-convergence and catastrophic metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.