MCMC non-convergence and catastrophic predictive_accuracy on 1.6.0 with official tfp-nightly dependency
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 294
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 47
Description
### Summary
Upgrading from `google-meridian==1.4.0` to `google-meridian==1.6.0` with the **official** dependency stack from [pyproject.toml](https://github.com/google/meridian/blob/v1.6.0/pyproject.toml) (`tfp-nightly==0.26.0.dev20260130`) causes complete MCMC non-convergence and nonsensical goodness-of-fit metrics, while the same training data, config, and priors work on 1.4.0.
### Environment
- Platform: linux aarch64 (Docker on Apple Silicon)
- Python: 3.11
- `google-meridian`: 1.6.0
- `tensorflow`: 2.20.0
- `tfp-nightly`: 0.26.0.dev20260130 (as required by pyproject.toml)
- `numpy`: 2.3.5 (< 2.4)
Working baseline (1.4.0):
- `google-meridian`: 1.4.0
- `tensorflow`: 2.20.0
- `tensorflow-probability`: 0.25.0 (stable; declared in [v1.4.0 pyproject.toml](https://github.com/google/meridian/blob/v1.4.0/pyproject.toml))
### Reproduction
National/geo MMM, `kpi_type=non_revenue`, no `revenue_per_kpi`, ROI priors via `PriorDistribution(roi_m=LogNormal(...))`, holdout_id mask, MCMC: 8 chains / 2000 adapt / 1000 burnin / 2000 keep.
Install official 1.6.0 stack:
```bash
pip install 'numpy<2.4' 'google-meridian==1.6.0'
pip freeze | grep -iE 'meridian|tensorflow|tfp'
```
Train model, then:
```python
from meridian.analysis import analyzer
from meridian.analysis.review import reviewer
analysis = analyzer.Analyzer(mmm)
print(analysis.predictive_accuracy(use_kpi=True).to_dataframe())
health = reviewer.ModelReviewer(mmm).run()
print(health.overall_status, health.health_score, health.summary_message)
```
### Actual results (1.6.0 + official tfp-nightly)
**Predictive accuracy (ALL geo model):**
| metric | geo_granularity | evaluation_set | value |
|--------|-----------------|----------------|-------|
| R_Squared | national | All Data | -68817.18 |
| MAPE | national | All Data | 62.83 |
| wMAPE | national | All Data | 64.02 |
**Health check:**
- `overall_status`: FAIL
- `health_score`: 0.0
- `max_r_hat` for `roi_m`: **338660163584.00**
- Message: "Failed: Model did not converge. Other checks were skipped."
### Expected results (1.4.0 + tensorflow-probability 0.25.0)
Same data and config on 1.4.0:
| metric | geo_granularity | evaluation_set | value |
|--------|-----------------|----------------|-------|
| R_Squared | national | All Data | 0.908 |
| MAPE | national | All Data | 0.051 |
| wMAPE | national | All Data | 0.053 |
### Notes
1. `tfp-nightly` is an **explicit** dependency in 1.5.3+ pyproject.toml, not an accidental resolver artifact.
2. 1.5.3 also declares the same `tfp-nightly` pin; we have not fully validated whether 1.5.3 official stack reproduces the issue.
3. Replacing `tfp-nightly` with `tensorflow-probability==0.25.0` after installing 1.6.0 is a possible workaround but violates declared dependencies and may break if 1.6.0 calls 0.26-only APIs.
Contributor guide
Research direction
Start by comparing the v1.6.0 and v1.4.0 pyproject.toml dependency declarations, then reproduce the reported training setup with the official v1.6.0 stack. Inspect analyzer.Analyzer.predictive_accuracy and reviewer.ModelReviewer outputs alongside MCMC convergence; done means the official dependency configuration no longer produces the reported non-convergence and catastrophic metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100