awslabs / awslabs/synthetically_engineered_evaluation_data

split_document.page_indices is always [0], even when the labelled content spans multiple pages

Open
#20 0 comments 0 reactions 1 assignee Claimed by @hayleypark View on GitHub
Dominant language
Python
Stars
9
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Summary

For multi-page generated documents, every ground-truth section is emitted with
`split_document.page_indices: [0]`, regardless of how many pages the document has
or which pages its labelled content actually appears on.

The field *values* look correct — the issue is purely the page-range metadata that
accompanies them. That makes the generated ground truth internally inconsistent: a
section asserts it covers only page 0 while carrying labels transcribed from pages
1–6.

## Impact

We consume SEED output as ground truth for evaluating an IDP pipeline, and this
breaks two things:

1. **Packet-splitting / section-boundary evaluation is unusable.** Scoring a
pipeline against this ground truth penalises it for splitting a 5-page
statement into 5 sections when the "truth" claims 1 section covering 1 page.
The resulting split-accuracy metric looks like a pipeline regression but is an
artifact of the metadata.

2. **Any consumer that renders a section by its page range shows the wrong pages.**
Our ground-truth editor honours `page_indices` faithfully, so a 4-page document
displays "Page 1 of 1" beside a transaction list that visibly continues onto
later pages. That reads as a viewer bug rather than a data one, which cost us a
while to track down.

## Reproduction

Generated bank statements (`Bank-Statement` class, 5 documents). For each, we
compared the emitted `page_indices` against (a) the PDF's real page count and
(b) which pages the ground-truth transactions actually appear on, by matching each
transaction's `description` against per-page extracted text (Unicode- and
whitespace-normalised, en-dash folded).

| document | PDF pages | GT transactions | GT transactions found on page… | `page_indices` emitted |
|---|---|---|---|---|
| doc_0001 | 3 | 23 | — | `[0]` |
| doc_0002 | 5 | 65 | 0 (33), 1 (21), 2 (6), 3 (5) | `[0]` |
| doc_0003 | 4 | 27 | 1, 2 | `[0]` |
| doc_0004 | 4 | 21 | 0 (1) | `[0]` |
| doc_0005 | 7 | 74 | **3 (66), 4 (7)** | `[0]` |

`doc_0005` is the clearest case: 74 transactions, none of which appear on page 0 —
66 are on page 3 and 7 on page 4 — yet the section claims `page_indices: [0]`.

`doc_0002` shows the spread most plainly: labelled content is distributed across
four of five pages.

(doc_0001 returned no positional matches; its PDF text layer did not extract
cleanly enough to locate rows, so it is listed for page-count context only.)

## Expected behaviour

`page_indices` should enumerate the 0-based pages the section's labelled content
actually spans — e.g. `[0, 1, 2, 3]` for doc_0002 — or, if a generated document is
intended to be a single logical section covering the whole file, the full page
range rather than just `[0]`.

Either convention works for us as long as it is accurate and documented. What we
cannot use is a fixed `[0]`, because it is indistinguishable from a genuine
single-page section.

## Environment

- Generated via the `idp-data-generator` extension in the GenAI IDP Accelerator,
which wraps `seed-data`.
- Documents: synthetic multi-page bank statements, one section each.
- Ground truth read from `baseline//sections/1/result.json`.

## Sample

`doc_0005` (7 pages), abbreviated:

```json
{
"document_class": { "type": "Bank-Statement" },
"split_document": { "page_indices": [0] },
"inference_result": {
"bank_name": "...",
"account_number": "...",
"transaction_details": [
{ "date": "...", "description": "Payoneer Credit - Apex Packaging", "...": "..." }
// 74 entries; content located on PDF pages 3 and 4
]
}
}
```

## Related

Separately, we noticed the generated field names do not match the document class
schema our pipeline was configured with — the generator emitted flat
`box_b_employer_identification_number`-style keys where our configuration expected
nested groups (`employer_info.ein`), with zero overlap between the two sets. That
may well be expected (author a matching config first), but if the intent is for
generated ground truth to be scoreable against an existing class schema, it is
worth confirming. Happy to open that as a separate issue if useful.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.