microsoft / microsoft/haste

[Bug]: Assessment Report returns a negative lower confidence bound for a building count

Open
#116 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
104
Forks
18
Avg merge
3d 12h
Merged PRs (30d)
50

Description

Summary

The Assessment Report's damaged-building population estimate can report a 95% confidence interval whose lower bound is negative, which is impossible for a count.

Observed

Spokane wildfire layer, DINOv2 ViT-B/14, 647 predicted damaged of 112,687 footprints, validated with the default 200-building sample (199 sure-labeled, 2 damaged):

Extrapolating to all 97,541 buildings with area > 50 m², we estimate 980 damaged buildings (1.0%) with a 95% CI of [-373, 2,334].

Field Value
N (buildings with area > 50 m²) 97,541
Sample size (n) 199
Damaged in sample (x) 2
Sample damage rate (p̂) 1.0%
Std. error of p̂ 0.007081
Estimated damaged (Ŷ) 980
95% CI [-373, 2,334]

Cause

The interval appears to be a normal (Wald) approximation, p̂ ± 1.96·SE, applied to x = 2. Wald is not valid at small counts — the usual guidance is np̂ ≥ 5 and n(1-p̂) ≥ 5, and here np̂ = 2. With p̂ = 0.01 and SE = 0.00708, the lower bound goes negative.

Suggested fix

  1. Use an interval valid at small counts — Wilson score or Clopper–Pearson — instead of Wald.
  2. Clamp reported bounds to [0, N] regardless of method.
  3. Warn in the report when x is small (say < 10) that the estimate is low-confidence.

Two related observations on the same report

  • The point estimate contradicts the model's own count. The report extrapolates from the sample to 980, while the model predicted 647. Both numbers appear in the product with equal authority and no explanation of why they differ or which to cite.
  • Precision/recall are reported as bare 100%. "Estimated recall 100.0% and precision 100.0%" comes from those same 2 positives, with no indication of how little that constrains the true value.

Related

The interval's width here is a direct consequence of the unstratified validation sample (#50) — a uniform sample of a 0.57%-prevalence class yielded only 2 positives. Fixing #50 improves the input, but this issue is about the estimator being invalid regardless of how the sample was drawn.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file or test is named; start at the Assessment Report code that computes the damaged-building confidence interval. Reproduce the Spokane example, replace the invalid small-count interval, constrain bounds to [0, N], and verify the low-count warning and related estimate presentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.