google / google/tfp-causalimpact

Determining the number of burn-in (warmup-steps): fixed rate in tfp-CausalImpact versus dynamic in the R implementation

Open
#38 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
166
Forks
19
PR merge metrics
No merged PRs in 30d

Description

Hello `tfp-causalimpact` team,

Thank you for maintaining this great package! I have some questions regarding how burn-in (or warm-up) steps are handled in tfp-causalimpact, particularly in comparison to the original R implementation.

From my understanding, in tfp-causalimpact, the number of burn-in steps is fixed at 1/9 of num_results, meaning the first 1/9th of the samples are discarded. In contrast, the R implementation adaptively determines the burn-in by analyzing the log-likelihood trajectory, using the following process:

- Compute the log-likelihood for each sample.
- Consider the last fraction of samples (e.g., the final 10%) and find a high quantile (e.g., the 90th percentile) of log-likelihood in this tail portion.
- Identify the earliest point in the chain where the log-likelihood exceeds this quantile, and drop all samples before this point.

This adaptive approach can, in some cases, lead to dropping the first 90% of samples, depending on the log-likelihood trajectory. I have a few questions about this:

1. Could you confirm if my understanding of the burn-in behavior in tfp-causalimpact and the R implementation is correct? If not, I’d appreciate any corrections or clarifications.
2. In tfp-causalimpact, is there a way to dynamically determine the number of warm-up steps, similar to the R implementation? If not, is this something that could be considered for future updates?
3. Even in the R implementation, I’m curious about the logic of looking at the last fraction of samples to determine the burn-in threshold. Wouldn’t it make more sense to use the first fraction (e.g., the first 20%) to determine the threshold? This might prevent discarding a large proportion of samples and ensure the point estimates and credible intervals are based on a sufficiently large set of samples.

Thank you for your time and for considering these questions. I look forward to hearing your insights!

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by locating tfp-causalimpact's warmup-step handling and comparing it with the R implementation's adaptive logic. Done would require an agreed behavior for dynamic warmup handling, with its expected results documented and validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, r
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.