ENH: better confidence intervals for weighted average or sum of count (non-gaussian) variables
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
Start with related issue #8138 and PR #8166, then review the cited methods from Ng et al., Dobson, Krishnamoorthy, and Swift. The issue names no files or tests; a complete contribution would need an agreed method scope and unit tests based on the worked example in Ng et al.
Written by the indexing model from the issue text.
Description
related to #8138 and PR #8166
standardized incidence rates are computed as population average based on subgroup rates and weights of group in population.
In the poisson case, this is the weighted sum of heterogeneous poisson rates.
The literature in this area build confidence intervals that are more accurate than simply assuming normal distribution of the weighted sum or average.
AFAICS, those method can be extended to other cases than Poisson (but I have only looked at references for poisson case)
Ng et al have a Monte Carlo with 20 methods, with formulas and references.
Dobson is one of the early articles, and they use a affine transformed univariate poisson confidence interval, where transformation is to match mean and variance.
further references for poisson are in articles, especially Ng.
Krishnamoorthy use MLS/MOVER confidence interval for weighted average, and for other poisson cases.
That's a good starting point to more general applications.
Dobson, Annette J., Kari Kuulasmaa, Ernst Eberle, and Josef Scherer. 1991. “Confidence Intervals for Weighted Sums of Poisson Parameters.” Statistics in Medicine 10 (3): 457–62. https://doi.org/10.1002/sim.4780100317.
Krishnamoorthy, K., Jie Peng, and Dan Zhang. 2016. “Modified Large Sample Confidence Intervals for Poisson Distributions: Ratio, Weighted Average, and Product of Means.” Communications in Statistics - Theory and Methods 45 (1): 83–97. https://doi.org/10.1080/03610926.2013.821486.
Ng, Hon Keung Tony, Giovanni Filardo, and Gang Zheng. 2008. “Confidence Interval Estimating Procedures for Standardized Incidence Rates.” Computational Statistics & Data Analysis 52 (7): 3501–16. https://doi.org/10.1016/j.csda.2007.11.004.
The Ng et al article has one worked example that could be used for unit tests of all methods.
another article Swift 2010 mentions that Ng et al have typographical errors in formulas, so they might not all be reliable.
Swift, Michael Bruce. 2010. “A Simulation Study Comparing Methods for Calculating Confidence Intervals for Directly Standardized Rates.” Computational Statistics & Data Analysis 54 (4): 1103–8. https://doi.org/10.1016/j.csda.2009.10.021.
compares a bootstrap method with selected methods from Ng 2008 article
- Dominant language
- Python
- Stars
- 11.6k
- Forks
- 3.6k
- Avg merge
- 7h 37m
- Merged PRs (30d)
- 96
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from statsmodels/statsmodels
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
statsmodels/statsmodels#10271 ·
-
type-bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
statsmodels/statsmodels#10269 ·
-
Documentation
Difficulty 2/5 1-3 hours Newbie friendliness 92/100
statsmodels/statsmodels#10266 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
statsmodels/statsmodels#9627 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
statsmodels/statsmodels#9293 · 1 comment ·
All issues in statsmodels/statsmodels
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100