awslabs / awslabs/synthetically_engineered_evaluation_data
split_document.page_indices is always [0], even when the labelled content spans multiple pages
- Dominant language
- Python
- Stars
- 9
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
For multi-page generated documents, every ground-truth section is emitted with
`split_document.page_indices: [0]`, regardless of how many pages the document has
or which pages its labelled content actually appears on.
The field *values* look correct — the issue is purely the page-range metadata that
accompanies them. That makes the generated ground truth internally inconsistent: a
section asserts it covers only page 0 while carrying labels transcribed from pages
1–6.
## Impact
We consume SEED output as ground truth for evaluating an IDP pipeline, and this
breaks two things:
1. **Packet-splitting / section-boundary evaluation is unusable.** Scoring a
pipeline against this ground truth penalises it for splitting a 5-page
statement into 5 sections when the "truth" claims 1 section covering 1 page.
The resulting split-accuracy metric looks like a pipeline regression but is an
artifact of the metadata.
2. **Any consumer that renders a section by its page range shows the wrong pages.**
Our ground-truth editor honours `page_indices` faithfully, so a 4-page document
displays "Page 1 of 1" beside a transaction list that visibly continues onto
later pages. That reads as a viewer bug rather than a data one, which cost us a
while to track down.
## Reproduction
Generated bank statements (`Bank-Statement` class, 5 documents). For each, we
compared the emitted `page_indices` against (a) the PDF's real page count and
(b) which pages the ground-truth transactions actually appear on, by matching each
transaction's `description` against per-page extracted text (Unicode- and
whitespace-normalised, en-dash folded).
| document | PDF pages | GT transactions | GT transactions found on page… | `page_indices` emitted |
|---|---|---|---|---|
| doc_0001 | 3 | 23 | — | `[0]` |
| doc_0002 | 5 | 65 | 0 (33), 1 (21), 2 (6), 3 (5) | `[0]` |
| doc_0003 | 4 | 27 | 1, 2 | `[0]` |
| doc_0004 | 4 | 21 | 0 (1) | `[0]` |
| doc_0005 | 7 | 74 | **3 (66), 4 (7)** | `[0]` |
`doc_0005` is the clearest case: 74 transactions, none of which appear on page 0 —
66 are on page 3 and 7 on page 4 — yet the section claims `page_indices: [0]`.
`doc_0002` shows the spread most plainly: labelled content is distributed across
four of five pages.
(doc_0001 returned no positional matches; its PDF text layer did not extract
cleanly enough to locate rows, so it is listed for page-count context only.)
## Expected behaviour
`page_indices` should enumerate the 0-based pages the section's labelled content
actually spans — e.g. `[0, 1, 2, 3]` for doc_0002 — or, if a generated document is
intended to be a single logical section covering the whole file, the full page
range rather than just `[0]`.
Either convention works for us as long as it is accurate and documented. What we
cannot use is a fixed `[0]`, because it is indistinguishable from a genuine
single-page section.
## Environment
- Generated via the `idp-data-generator` extension in the GenAI IDP Accelerator,
which wraps `seed-data`.
- Documents: synthetic multi-page bank statements, one section each.
- Ground truth read from `baseline//sections/1/result.json`.
## Sample
`doc_0005` (7 pages), abbreviated:
```json
{
"document_class": { "type": "Bank-Statement" },
"split_document": { "page_indices": [0] },
"inference_result": {
"bank_name": "...",
"account_number": "...",
"transaction_details": [
{ "date": "...", "description": "Payoneer Credit - Apex Packaging", "...": "..." }
// 74 entries; content located on PDF pages 3 and 4
]
}
}
```
## Related
Separately, we noticed the generated field names do not match the document class
schema our pipeline was configured with — the generator emitted flat
`box_b_employer_identification_number`-style keys where our configuration expected
nested groups (`employer_info.ein`), with zero overlap between the two sets. That
may well be expected (author a matching config first), but if the intent is for
generated ground truth to be scoreable against an existing class schema, it is
worth confirming. Happy to open that as a separate issue if useful.
Contributor guide
Assessment
This issue has not been assessed yet.