anthropics / anthropics/financial-services

extract_numbers.py: a CORRECT multi-year deck reports 5 'high severity' inconsistencies — grouping ignores the period

未关闭
#339 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
34.8k
派生
5.2k
平均合并
2 小时 2 分钟
30 天内合并 PR
3

描述

## Summary

`find_inconsistencies` in `skills/ib-check-deck/scripts/extract_numbers.py`
groups figures by **category alone** and flags anything more than 5% from the
largest group. A deck carrying FY2023, FY2024 and FY2025E revenue is therefore
self-contradictory by construction — which is every deck with a financial
history.

Measured on a four-line deck: **a correct deck produced 5 "high severity"
inconsistencies**, against **1** for a deck containing a genuine contradiction.
Precision ≈ 1/6. The true finding is indistinguishable from the noise.

## Reproduce

`clean.md` — correct, no contradiction:
```markdown
## Slide 2 — Historical Performance
Revenue grew from $100.0 million in FY2023 to $120.0 million in FY2024.
EBITDA was $20.0 million in FY2023 and $25.0 million in FY2024.
## Slide 5 — Projections
Revenue of $145.0 million is forecast for FY2025E.
```

`planted.md` — one genuine contradiction (FY2024 revenue stated twice):
```markdown
## Slide 2 — Historical Performance
Revenue was $120.0 million in FY2024.
## Slide 7 — Transaction Summary
FY2024 revenue of $135.0 million supports the valuation.
```

```
$ python extract_numbers.py clean.md --check -> 5 inconsistencies (all severity "high")
$ python extract_numbers.py planted.md --check -> 1 inconsistency
```

On the clean deck it reports `revenue $100.0million vs $120.0million`,
`vs $145.0million`, and `ebitda $20.0million vs $25.0million` — all correct
figures for different periods. (One of the five is `2023t`, the separate
parsing defect filed alongside this.)

## Cause

```python
# lines ~192–210
for num in numbers:
if num.category != 'other':
by_category[num.category].append(num) # period is discarded
...
if diff_pct < 0.05: # 5% tolerance on VALUE only
```

Nothing distinguishes "the same metric stated twice inconsistently" from "the
same metric across three periods".

## Suggested fix

Key the comparison on **(category, period)**, not category alone:

- parse a period marker near each figure — `FY2024`, `FY24`, `2025E`, `2023A`;
- attribute a figure to the period that **follows** it before the next figure,
falling back to the preceding one. Nearest-by-distance is wrong on the
commonest sentence there is: in *"EBITDA was $20.0m in FY2023 and $25.0m in
FY2024"*, `25.0` sits 13 chars after `FY2023` and 18 before `FY2024`, so
distance alone assigns both figures to FY2023 and re-creates the false
positive;
- only report when two values share a category **and** a period;
- skip figures with no resolvable period rather than pooling them.

With that change the two fixtures above give **0** and **1** respectively.

Happy to open a PR.

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。