anthropics / anthropics/financial-services

extract_numbers.py: a CORRECT multi-year deck reports 5 'high severity' inconsistencies — grouping ignores the period

Đang mở
#339 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
34.8k
Fork
5.2k
Merge trung bình
2 giờ 2 phút
Pull request đã merge (30 ngày)
3

Mô tả

## Summary

`find_inconsistencies` in `skills/ib-check-deck/scripts/extract_numbers.py`
groups figures by **category alone** and flags anything more than 5% from the
largest group. A deck carrying FY2023, FY2024 and FY2025E revenue is therefore
self-contradictory by construction — which is every deck with a financial
history.

Measured on a four-line deck: **a correct deck produced 5 "high severity"
inconsistencies**, against **1** for a deck containing a genuine contradiction.
Precision ≈ 1/6. The true finding is indistinguishable from the noise.

## Reproduce

`clean.md` — correct, no contradiction:
```markdown
## Slide 2 — Historical Performance
Revenue grew from $100.0 million in FY2023 to $120.0 million in FY2024.
EBITDA was $20.0 million in FY2023 and $25.0 million in FY2024.
## Slide 5 — Projections
Revenue of $145.0 million is forecast for FY2025E.
```

`planted.md` — one genuine contradiction (FY2024 revenue stated twice):
```markdown
## Slide 2 — Historical Performance
Revenue was $120.0 million in FY2024.
## Slide 7 — Transaction Summary
FY2024 revenue of $135.0 million supports the valuation.
```

```
$ python extract_numbers.py clean.md --check -> 5 inconsistencies (all severity "high")
$ python extract_numbers.py planted.md --check -> 1 inconsistency
```

On the clean deck it reports `revenue $100.0million vs $120.0million`,
`vs $145.0million`, and `ebitda $20.0million vs $25.0million` — all correct
figures for different periods. (One of the five is `2023t`, the separate
parsing defect filed alongside this.)

## Cause

```python
# lines ~192–210
for num in numbers:
if num.category != 'other':
by_category[num.category].append(num) # period is discarded
...
if diff_pct < 0.05: # 5% tolerance on VALUE only
```

Nothing distinguishes "the same metric stated twice inconsistently" from "the
same metric across three periods".

## Suggested fix

Key the comparison on **(category, period)**, not category alone:

- parse a period marker near each figure — `FY2024`, `FY24`, `2025E`, `2023A`;
- attribute a figure to the period that **follows** it before the next figure,
falling back to the preceding one. Nearest-by-distance is wrong on the
commonest sentence there is: in *"EBITDA was $20.0m in FY2023 and $25.0m in
FY2024"*, `25.0` sits 13 chars after `FY2023` and 18 before `FY2024`, so
distance alone assigns both figures to FY2023 and re-creates the false
positive;
- only report when two values share a category **and** a period;
- skip figures with no resolvable period rather than pooling them.

With that change the two fixtures above give **0** and **1** respectively.

Happy to open a PR.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Hướng nghiên cứu

Start in skills/ib-check-deck/scripts/extract_numbers.py, around find_inconsistencies and the grouping logic at lines ~192–210. Run the script with --check on the clean.md and planted.md examples; done means the clean fixture reports 0 inconsistencies and the planted fixture reports 1, while figures from different periods are not compared.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
tooling
Loại issue
Lỗi
Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Đặc tả rõ ràng
Mức phù hợp với người mới
70/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.