ENH: better confidence intervals for weighted average or sum of count (non-gaussian) variables

Open
#8,174 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
data

Research direction

Start with related issue #8138 and PR #8166, then review the cited methods from Ng et al., Dobson, Krishnamoorthy, and Swift. The issue names no files or tests; a complete contribution would need an agreed method scope and unit tests based on the worked example in Ng et al.

Written by the indexing model from the issue text.

Description

comp-stats type-enh

related to #8138 and PR #8166

standardized incidence rates are computed as population average based on subgroup rates and weights of group in population.
In the poisson case, this is the weighted sum of heterogeneous poisson rates.

The literature in this area build confidence intervals that are more accurate than simply assuming normal distribution of the weighted sum or average.
AFAICS, those method can be extended to other cases than Poisson (but I have only looked at references for poisson case)

Ng et al have a Monte Carlo with 20 methods, with formulas and references.
Dobson is one of the early articles, and they use a affine transformed univariate poisson confidence interval, where transformation is to match mean and variance.

further references for poisson are in articles, especially Ng.

Krishnamoorthy use MLS/MOVER confidence interval for weighted average, and for other poisson cases.
That's a good starting point to more general applications.

Dobson, Annette J., Kari Kuulasmaa, Ernst Eberle, and Josef Scherer. 1991. “Confidence Intervals for Weighted Sums of Poisson Parameters.” Statistics in Medicine 10 (3): 457–62. https://doi.org/10.1002/sim.4780100317.

Krishnamoorthy, K., Jie Peng, and Dan Zhang. 2016. “Modified Large Sample Confidence Intervals for Poisson Distributions: Ratio, Weighted Average, and Product of Means.” Communications in Statistics - Theory and Methods 45 (1): 83–97. https://doi.org/10.1080/03610926.2013.821486.

Ng, Hon Keung Tony, Giovanni Filardo, and Gang Zheng. 2008. “Confidence Interval Estimating Procedures for Standardized Incidence Rates.” Computational Statistics & Data Analysis 52 (7): 3501–16. https://doi.org/10.1016/j.csda.2007.11.004.

The Ng et al article has one worked example that could be used for unit tests of all methods.
another article Swift 2010 mentions that Ng et al have typographical errors in formulas, so they might not all be reliable.

Swift, Michael Bruce. 2010. “A Simulation Study Comparing Methods for Calculating Confidence Intervals for Directly Standardized Rates.” Computational Statistics & Data Analysis 54 (4): 1103–8. https://doi.org/10.1016/j.csda.2009.10.021.

compares a bootstrap method with selected methods from Ng 2008 article

Dominant language
Python
Stars
11.6k
Forks
3.6k
Avg merge
7h 37m
Merged PRs (30d)
96

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from statsmodels/statsmodels

All issues in statsmodels/statsmodels

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.