awslabs / awslabs/synthetically_engineered_evaluation_data

seed-policy: Proposal for a policy-grounded IDP benchmark extension to SEED

Open
#28 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Summary

This issue proposes **`seed-policy`**, a new library that extends `seed-data` with a policy-grounded, scenario-driven document generation layer for IDP rule-validation benchmarks.

The library wraps SEED's `Generator` and `Schema` with a `PolicyDomain` abstraction — the first extension mechanism for SEED. Health insurance is the reference implementation; the design is fully generic across any policy-and-claim domain (life, auto, dental, property, etc.).

cc @sromoam — tagging you for review.

## The gap SEED currently has

`seed-data` generates realistic PDFs from a JSON Schema but has no concept of:
- A **policy document** as grounding context for generation
- A **scenario** with *by-construction known ground truth* (outcome recorded at generation time, never in the document — required for fair IDP benchmarks)
- **Domain-specific validators** (CPT code validity, financial math, section completeness)
- **Clean document / ground-truth separation** — the generated claim must contain zero outcome hints

## What `seed-policy` adds

One new abstraction — `PolicyDomain` — that a user implements once to define a new document domain. Everything else (doc-gen loop, augmentation, batch, packet builder) is inherited unchanged from SEED.

```python
gen = PolicyGenerator(LifeInsuranceDomain(), data_model="gpt-oss", critic_model="haiku")
result = gen.generate_benchmark(
policy_dir="data/life_policies",
scenarios=["straight_valid", "contestable_misrep"],
n_policies=10,
package=True,
)
```

Key design decisions:
- **Item selection is LLM-based and policy-grounded** — reads the actual policy text to select a genuinely applicable excluded service/event for each scenario
- **Three-tier token budget** (full text → scenario-focused excerpt → structured summary) keeps generation cost low for long policy PDFs
- **Policy evidence in ground truth** — every `GeneratedClaim` carries a verbatim policy quote justifying the expected outcome
- **`PolicyGenerator` composes `Generator`** — builds its own extended Strands graph reusing SEED's doc-gen patterns; `seed-data` is a runtime dependency for public types only

## Full API design spec

The spec has been through 4 rounds of adversarial review (correctness + domain-agnosticism pass):

📄 [api_design_spec_policy_version.md](https://gist.github.com/saeidesh/52d0d6d85e1cb6f3bf5c24e8ce8764b7)

## Questions for the SEED team

1. Does this extension model fit the direction you have in mind for `seed-data`?
2. Is there a better integration point than reimplementing the doc-gen graph in `seed_policy/orchestrate.py`? (We avoided importing `seed_data.stages` since it is not part of the public API)
3. Would you consider `seed-policy` as a first-party extension inside the `seed-data` repo, or is a separate package the preferred path?

Happy to walk through the design in more detail.

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked api_design_spec_policy_version.md and the existing seed-data Generator and Schema abstractions. Then examine the proposed seed_policy/orchestrate.py integration point and determine whether the extension model and package boundary answer the three questions; done means the SEED team has agreed on the integration path and scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.