awslabs / awslabs/synthetically_engineered_evaluation_data
seed-policy: Proposal for a policy-grounded IDP benchmark extension to SEED
- Dominant language
- Python
- Stars
- 9
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
This issue proposes **`seed-policy`**, a new library that extends `seed-data` with a policy-grounded, scenario-driven document generation layer for IDP rule-validation benchmarks.
The library wraps SEED's `Generator` and `Schema` with a `PolicyDomain` abstraction — the first extension mechanism for SEED. Health insurance is the reference implementation; the design is fully generic across any policy-and-claim domain (life, auto, dental, property, etc.).
cc @sromoam — tagging you for review.
## The gap SEED currently has
`seed-data` generates realistic PDFs from a JSON Schema but has no concept of:
- A **policy document** as grounding context for generation
- A **scenario** with *by-construction known ground truth* (outcome recorded at generation time, never in the document — required for fair IDP benchmarks)
- **Domain-specific validators** (CPT code validity, financial math, section completeness)
- **Clean document / ground-truth separation** — the generated claim must contain zero outcome hints
## What `seed-policy` adds
One new abstraction — `PolicyDomain` — that a user implements once to define a new document domain. Everything else (doc-gen loop, augmentation, batch, packet builder) is inherited unchanged from SEED.
```python
gen = PolicyGenerator(LifeInsuranceDomain(), data_model="gpt-oss", critic_model="haiku")
result = gen.generate_benchmark(
policy_dir="data/life_policies",
scenarios=["straight_valid", "contestable_misrep"],
n_policies=10,
package=True,
)
```
Key design decisions:
- **Item selection is LLM-based and policy-grounded** — reads the actual policy text to select a genuinely applicable excluded service/event for each scenario
- **Three-tier token budget** (full text → scenario-focused excerpt → structured summary) keeps generation cost low for long policy PDFs
- **Policy evidence in ground truth** — every `GeneratedClaim` carries a verbatim policy quote justifying the expected outcome
- **`PolicyGenerator` composes `Generator`** — builds its own extended Strands graph reusing SEED's doc-gen patterns; `seed-data` is a runtime dependency for public types only
## Full API design spec
The spec has been through 4 rounds of adversarial review (correctness + domain-agnosticism pass):
📄 [api_design_spec_policy_version.md](https://gist.github.com/saeidesh/52d0d6d85e1cb6f3bf5c24e8ce8764b7)
## Questions for the SEED team
1. Does this extension model fit the direction you have in mind for `seed-data`?
2. Is there a better integration point than reimplementing the doc-gen graph in `seed_policy/orchestrate.py`? (We avoided importing `seed_data.stages` since it is not part of the public API)
3. Would you consider `seed-policy` as a first-party extension inside the `seed-data` repo, or is a separate package the preferred path?
Happy to walk through the design in more detail.
Contributor guide
Research direction
Start by reading the linked api_design_spec_policy_version.md and the existing seed-data Generator and Schema abstractions. Then examine the proposed seed_policy/orchestrate.py integration point and determine whether the extension model and package boundary answer the three questions; done means the SEED team has agreed on the integration path and scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100