stan-dev / stan-dev/loo

Add support for Jacobian adjustment

Open
#265 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
157
Forks
38
Avg merge
4d 16h
Merged PRs (30d)
2

Description

Related to discussion in a Stan discourse post

ROS book and some Stan discourse answers show how to make Jacobian adjustment for elpd_loo values if different models use different transformations of the continuous target. I had tested that a simple approach of adjusting the pointwise values as

loo_2_with_jacobian <- loo_2
loo_2_with_jacobian$pointwise[,1] <- loo_2_with_jacobian$pointwise[,1] - log(2*sqrt(data$y))

works when comparing two models. In case of more than two models, the summary values in $estimates slot are used to sort the models, which causes problems if the order is different after Jacobian adjustment. A quick fix is to update the summary, too

loo_2_with_jacobian$estimates["elpd_loo",] <- loo:::table_of_estimates(loo_2_with_jacobian$pointwise[,"elpd_loo", drop=FALSE])

It would be better to make a function that would properly update all the slots, or probably even better to add jacobian argument to loo() function.

This update would not be big, doesn't cause backward compatibility issues, and would make the life of users easier

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the loo() output structure, especially the $pointwise and $estimates slots, and inspect the referenced table_of_estimates() helper. Define how Jacobian adjustment should be supplied and ensure all affected summaries stay consistent, including model ordering when comparing more than two models.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.