[Bug]: Assessment Report returns a negative lower confidence bound for a building count
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 104
- Forks
- 18
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 50
Description
Summary
The Assessment Report's damaged-building population estimate can report a 95% confidence interval whose lower bound is negative, which is impossible for a count.
Observed
Spokane wildfire layer, DINOv2 ViT-B/14, 647 predicted damaged of 112,687 footprints, validated with the default 200-building sample (199 sure-labeled, 2 damaged):
Extrapolating to all 97,541 buildings with area > 50 m², we estimate 980 damaged buildings (1.0%) with a 95% CI of [-373, 2,334].
| Field | Value |
|---|---|
| N (buildings with area > 50 m²) | 97,541 |
| Sample size (n) | 199 |
| Damaged in sample (x) | 2 |
| Sample damage rate (p̂) | 1.0% |
| Std. error of p̂ | 0.007081 |
| Estimated damaged (Ŷ) | 980 |
| 95% CI | [-373, 2,334] |
Cause
The interval appears to be a normal (Wald) approximation, p̂ ± 1.96·SE, applied to x = 2. Wald is not valid at small counts — the usual guidance is np̂ ≥ 5 and n(1-p̂) ≥ 5, and here np̂ = 2. With p̂ = 0.01 and SE = 0.00708, the lower bound goes negative.
Suggested fix
- Use an interval valid at small counts — Wilson score or Clopper–Pearson — instead of Wald.
- Clamp reported bounds to
[0, N]regardless of method. - Warn in the report when
xis small (say< 10) that the estimate is low-confidence.
Two related observations on the same report
- The point estimate contradicts the model's own count. The report extrapolates from the sample to 980, while the model predicted 647. Both numbers appear in the product with equal authority and no explanation of why they differ or which to cite.
- Precision/recall are reported as bare 100%. "Estimated recall 100.0% and precision 100.0%" comes from those same 2 positives, with no indication of how little that constrains the true value.
Related
The interval's width here is a direct consequence of the unstratified validation sample (#50) — a uniform sample of a 0.57%-prevalence class yielded only 2 positives. Fixing #50 improves the input, but this issue is about the estimator being invalid regardless of how the sample was drawn.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file or test is named; start at the Assessment Report code that computes the damaged-building confidence interval. Reproduce the Spokane example, replace the invalid small-count interval, constrain bounds to [0, N], and verify the low-count warning and related estimate presentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100