stan-dev / stan-dev/loo

Detect non-finite `log_lik` (`Inf` / `NaN`) and explicitely inform the user

Open
#386 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
157
Forks
38
Avg merge
4d 16h
Merged PRs (30d)
2

Description

Summary

If the pointwise log-likelihood matrix contains +Inf or NaN, loo::loo() currently completes without a clear error/warning and can return NA for elpd_loo.

Proposal

Before (or while) computing PSIS-LOO, check the log_lik matrix for non-finite values and message this clearly to the user.

Reproducible example

library(loo)

set.seed(1)
ll <- matrix(rnorm(400 * 20, -1, 0.5), 400, 20)
ll[1:5, 1] <- Inf
loo(ll)$estimates  # elpd_loo is NA; no clear message that log_lik had +Inf
#          Estimate SE
# elpd_loo       NA NA
# p_loo          NA NA
# looic          NA NA

Context

Seen with truncated-normal log_lik from brms, where the truncation normalizer can underflow:
https://discourse.mc-stan.org/t/difficulty-adding-loo-criterion-to-longitudinal-truncated-normal-model/41447/12

Corresponding brms Issue: https://github.com/paul-buerkner/brms/issues/1899

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the loo::loo() entry point and trace where the log_lik matrix is validated or passed into PSIS-LOO. Use the reproducible matrix containing +Inf to verify that non-finite values are clearly reported instead of silently producing NA estimates. Done means the user-facing behavior identifies the invalid log_lik values and the existing example no longer completes without that information.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.