google / google/meridian

Demo code not converging on L4 or A100 GPU

Open
#1,446 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.5k
Forks
294
Avg merge
1d 21h
Merged PRs (30d)
47

Description

**Description**
I am encountering a convergence issue when running the [`Meridian_Getting_Started.ipynb`](https://github.com/google/meridian/blob/main/demo/Meridian_Getting_Started.ipynb) demo. The code executes correctly and the model converges on a **T4 GPU**, but fails to converge when running the exact same environment and code on **L4** or **A100** GPUs.

**To Reproduce**

```python
%%time
mmm.sample_prior(500)
mmm.sample_posterior(
n_chains=10, n_adapt=2000, n_burnin=500, n_keep=1000, seed=0
)

```

**Expected Behavior**
The model should converge similarly to how it performs on a T4 GPU.

**Actual Behavior**
On L4/A100 GPUs, I receive PTXAS warnings during compilation, followed by a total convergence failure.

**Logs & Output**
*Compilation Warnings:*

```
2026-02-01 19:40:50.474780: I external/local_xla/xla/stream_executor/cuda/subprocess_compilation.cc:346] ptxas warning : Registers are spilled to local memory in function 'gemm_fusion_dot_427', 8 bytes spill stores, 8 bytes spill loads

2026-02-01 19:41:53.997341: I external/local_xla/xla/stream_executor/cuda/subprocess_compilation.cc:346] ptxas warning : Registers are spilled to local memory in function 'input_concatenate_fusion_36', 4 bytes spill stores, 4 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'input_concatenate_fusion_37', 4 bytes spill stores, 4 bytes spill loads

```

*Model Reviewer Output:*

```python
reviewer.ModelReviewer(mmm).run()
```

```
========================================
Model Quality Checks
========================================
Overall Status: FAIL
Summary: Failed: Model did not converge. Other checks were skipped.

Check Results:
----------------------------------------
Convergence Check:
Status: FAIL
Recommendation: The model hasn't converged, and the `max_r_hat` for parameter `mu_t` is inf. We recommend increasing MCMC iterations or investigating model misspecification (e.g., priors, multicollinearity) before proceeding.
```

**Environment**

* **Hardware:** L4 or A100 GPU (Issue present), T4 (Issue absent)
* **Meridian Version:** 1.5
* **TensorFlow Version:** 2.19.0

**Attempted Solutions**
I attempted the following fixes (suggested by Gemini) to mitigate potential XLA/OneDNN issues, but neither solved the problem:

1. Disabling XLA Auto JIT:
`os.environ['TF_XLA_FLAGS'] = '--tf_xla_auto_jit=0'`
2. Disabling OneDNN:
`os.environ['TF_ENABLE_ONEDNN_OPTS'] = '0'`

Contributor guide

Open the contributing guide

Research direction

Start with demo/Meridian_Getting_Started.ipynb and run the sample_prior and sample_posterior calls on T4, L4, and A100 hardware using Meridian 1.5 and TensorFlow 2.19.0. Compare the PTXAS warnings and ModelReviewer output, then determine what change is needed for the model to converge on L4 and A100 as it does on T4.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.