awslabs / awslabs/synthetically_engineered_evaluation_data
SEED Unification: Integrate structured data generation from seed-tabular
- Dominant language
- Python
- Stars
- 9
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Merge `seed-synthetically-engineered-evaluation-data-from-discovery` (branch `hp/agentic-redesign`) into this repo, expanding `seed-data` to support **structured data generation** (CSV/Parquet/Excel) alongside the existing **document generation** (PDFs).
The two projects are the front half and back half of the same pipeline:
- **seed-tabular** (being merged in): ingests 6 input types → converges to one schema object → emits structured data
- **seed-data** (this repo): takes a schema + guidance → emits rendered PDFs with critique loops and augmentation
After unification, one package handles: `INPUT → InferredSchema → structured data OR documents`.
## Key Constraints
- **No breaking changes.** Existing `pip install seed-data`, `from seed_data import Generator`, `seed-data --schema-dir ...` all continue to work at every milestone.
- **Additive only.** New subpackages (`seed_data.schema`, `seed_data.ingest`, `seed_data.structured`, `seed_data.evaluation`) are added; nothing is renamed or removed.
- **Optional structured deps.** `pandas`/`numpy`/`scipy` live in `[structured]` extra — base install stays lean.
## Architecture After Unification
```
seed-data ingest → InferredSchema (on disk as JSON)
seed-data generate-structured → CSV/Parquet/Excel
seed-data generate-documents → PDFs + JSON labels
seed-data run --output ... → end-to-end dispatch
```
## Milestones
### Milestone 1: Unified Schema Layer (~1 week)
- [ ] Convert `schema.py` → `schema/` subpackage (legacy `Schema` class preserved)
- [ ] Port `InferredSchema`, `EntitySchema`, `FieldDefinition`, `DistributionSpec`, `RelationshipDefinition` from seed-tabular
- [ ] Add `FieldDefinition.children` for nested object support
- [ ] Add `EntitySchema.generation_guidance` and `EntitySchema.reference_samples`
- [ ] Implement `schema/io.py`: `InferredSchema ↔ JSON Schema` converters
- [ ] Round-trip test all 17 built-in schemas through the converter
- [ ] All existing tests pass unchanged
### Milestone 2: Ingest + Structured Data Generation (~2 weeks)
- [ ] Add `[structured]` optional dependency group (pandas, numpy, scipy, openpyxl)
- [ ] Port schema extraction agent → `seed_data.ingest/`
- [ ] Port graph pipeline + bulk generation → `seed_data.structured/`
- [ ] Port distribution generators → `seed_data.structured/distributions/`
- [ ] Port post-processing pipeline → `seed_data.structured/postprocessing/`
- [ ] Port evaluation metrics → `seed_data.evaluation/`
- [ ] Port Jinja2 prompts
- [ ] Wire CLI: `seed-data ingest` and `seed-data generate-structured`
- [ ] Wire Python API: `from seed_data import ingest, generate_structured`
- [ ] Unit tests for all ported modules (no Bedrock required)
- [ ] Integration tests (Bedrock required)
### Milestone 3: Wire Document Generation to Unified Schema (~1 week)
- [ ] Implement `schema/adapter.py`: `InferredSchema → (json_schema_dict, guidance, samples)`
- [ ] Wire CLI: `seed-data generate-documents `
- [ ] Parametrized roundtrip test: all 17 schemas through adapter preserve field names/types
- [ ] Integration test: `ingest → InferredSchema → generate-documents → PDF`
### Milestone 4: End-to-End Integration (~1 week)
- [ ] Implement `seed-data run` unified dispatch command
- [ ] Cross-modality evaluation: LLM critique for structured, quantitative metrics for doc labels
- [ ] Cross-modality consistency test: same schema → both outputs have same fields
- [ ] Python API: `from seed_data import run`
### Milestone 5: Polish & Open-Source Release (~1 week)
- [ ] GitHub Actions CI (lint + test for both dep groups)
- [ ] Update README with structured data section and unified architecture
- [ ] Update docs (CLI usage, Python API, guides)
- [ ] Audit for internal references (URLs, model IDs, credentials)
- [ ] CONTRIBUTING.md: how to add input modes, output modalities, schema types
- [ ] Fresh-clone verification for both modalities
## Detailed Plans
Full implementation steps and testing plans for each milestone are in `docs/planning/` on branch `hp/seed-unification`:
- `docs/planning/MILESTONE_1_UNIFIED_SCHEMA.md`
- `docs/planning/MILESTONE_2_INGEST_AND_STRUCTURED.md`
- `docs/planning/MILESTONE_3_DOCUMENT_GENERATION_WIRING.md`
- `docs/planning/MILESTONE_4_END_TO_END_INTEGRATION.md`
- `docs/planning/MILESTONE_5_POLISH_AND_RELEASE.md`
## Source
Porting from: `seed-synthetically-engineered-evaluation-data-from-discovery` branch `hp/agentic-redesign`
Plan reference: `../seed-ideas/SEED_UNIFICATION_PLAN.md`
Contributor guide
Assessment
This issue has not been assessed yet.