awslabs / awslabs/synthetically_engineered_evaluation_data

seed-policy: Proposal for a policy-grounded IDP benchmark extension to SEED

Open
#27 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Summary

This issue proposes **`seed-policy`**, a new library that extends `seed-data` with a policy-grounded, scenario-driven document generation layer for IDP rule-validation benchmarks.

The library wraps SEED's `Generator` and `Schema` with a `PolicyDomain` abstraction — the first extension mechanism for SEED. Health insurance is the reference implementation; the design is fully generic across any policy-and-claim domain (life, auto, dental, property, etc.).

## The gap SEED currently has

`seed-data` generates realistic PDFs from a JSON Schema but has no concept of:
- A **policy document** as grounding context for generation
- A **scenario** with *by-construction known ground truth* (outcome recorded at generation time, never in the document — required for fair IDP benchmarks)
- **Domain-specific validators** (CPT code validity, financial math, section completeness)
- **Clean document / ground-truth separation** — the generated claim must contain zero outcome hints

## What `seed-policy` adds

One new abstraction — `PolicyDomain` — that a user implements once to define a new document domain. Everything else (doc-gen loop, augmentation, batch, packet builder) is inherited unchanged from SEED.

```python
gen = PolicyGenerator(LifeInsuranceDomain(), data_model="gpt-oss", critic_model="haiku")
result = gen.generate_benchmark(
policy_dir="data/life_policies",
scenarios=["straight_valid", "contestable_misrep"],
n_policies=10,
package=True,
)
```

Key design decisions:
- **Item selection is LLM-based and policy-grounded** — reads the actual policy text to select a genuinely applicable excluded service/event for each scenario
- **Three-tier token budget** (full text → scenario-focused excerpt → structured summary) keeps generation cost low for long policy PDFs
- **Policy evidence in ground truth** — every `GeneratedClaim` carries a verbatim policy quote justifying the expected outcome
- **`PolicyGenerator` composes `Generator`** — builds its own extended Strands graph reusing SEED's doc-gen patterns; `seed-data` is a runtime dependency for public types only

## Full API design spec

The spec has been through 4 rounds of adversarial review (correctness + domain-agnosticism pass):

📄 [api_design_spec_policy_version.md](https://gist.github.com/saeidesh/52d0d6d85e1cb6f3bf5c24e8ce8764b7)

## Questions for the SEED team

1. Does this extension model fit the direction you have in mind for `seed-data`?
2. Is there a better integration point than reimplementing the doc-gen graph in `seed_policy/orchestrate.py`? (We avoided importing `seed_data.stages` since it's not part of the public API)
3. Would you consider `seed-policy` as a first-party extension inside the `seed-data` repo, or is a separate package the preferred path?

Happy to walk through the design in more detail — let me know what questions you have.

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked api_design_spec_policy_version.md and the proposal's references to seed_policy/orchestrate.py, then compare the proposed PolicyDomain and PolicyGenerator boundaries with seed-data's public API. Done means the SEED team has agreed on whether this belongs in the repository or a separate package and identified an accepted integration point; the issue does not yet define an implementation task.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.