Validate supporting_quotes field against source document content during extraction

Problem

Currently, when the LLM extracts politician properties during enrichment, it returns a proof field (stored as supporting_quotes in the database) that should contain a quote from the source document. However, we don't validate that this proof text actually exists in the document, which could lead to:

1. Hallucinated proofs: The LLM might generate plausible-sounding quotes that don't exist in the source
2. Paraphrased proofs: The LLM might summarize instead of quoting directly
3. Data quality issues: Invalid proofs make it harder to verify extractions and reduce trust in the system

Current Flow

The enrichment pipeline (poliloom/enrichment.py) works as follows:

1. Fetch & Process (_fetch_and_extract_from_page):

   • Fetches Wikipedia page and archives it
   • Converts HTML to plain text: content = " ".join(soup.get_text().split())
   • Passes this content to extraction functions

2. Extract Properties (extract_properties_generic and extract_two_stage_generic):

   • Calls OpenAI API with the content string
   • Returns Pydantic models with proof fields:
     • ExtractedProperty (for dates)
     • FreeFormPosition → ExtractedPosition (two-stage)
     • FreeFormBirthplace → ExtractedBirthplace (two-stage)
     • FreeFormCitizenship → ExtractedCitizenship (two-stage)

3. Store Data (store_extracted_data):

   • Saves to database with supporting_quotes field

Proposed Solution

Add model_validator to all Pydantic models that have a proof field to validate the proof appears in the source content.

Implementation Approach

Option 1: Validation Context (Recommended)

Use Pydantic's validation context to pass the source content during model parsing:

from pydantic import BaseModel, model_validator
from typing import Optional

class ExtractedProperty(BaseModel):
    type: PropertyType
    value: str
    proof: str
    
    @field_validator("value")
    @classmethod
    def validate_date_value(cls, v: str) -> str:
        return WikidataDate.validate_date_format(v)
    
    @model_validator(mode='after')
    def validate_proof_in_content(self) -> 'ExtractedProperty':
        """Validate that proof text appears in the source content."""
        # Access context passed during parsing
        if hasattr(self, 'model_config') and 'content' in self.model_config.get('context', {}):
            content = self.model_config['context']['content']
            if self.proof not in content:
                raise ValueError(
                    f"Proof text not found in source document: '{self.proof[:100]}...'"
                )
        return self

Then modify extract_properties_generic to pass context:

response = await openai_client.responses.parse(
    model="gpt-5",
    input=[
        {"role": "system", "content": config.system_prompt},
        {"role": "user", "content": user_prompt},
    ],
    text_format=config.result_model,
    reasoning={"effort": "minimal"},
    # Pass content in validation context
    validation_context={"content": content}
)

Option 2: Post-Extraction Validation

Add a validation function that runs after extraction but before storage:

def validate_proofs(extracted_items: List[Any], content: str) -> List[Any]:
    """Filter out items whose proof doesn't appear in content."""
    valid_items = []
    for item in extracted_items:
        if hasattr(item, 'proof') and item.proof in content:
            valid_items.append(item)
        else:
            logger.warning(f"Filtered out item with invalid proof: {item.proof[:100]}")
    return valid_items

Call this in _fetch_and_extract_from_page after extraction:

date_properties = validate_proofs(date_properties, content) if date_properties else None
positions = validate_proofs(positions, content) if positions else None
# etc.

Models to Update

All models with proof fields:

1. Direct extraction (extract_properties_generic):

   • ExtractedProperty (line 65)

2. Two-stage extraction (extract_two_stage_generic):

   • FreeFormPosition (line 107)
   • FreeFormBirthplace (line 132)
   • FreeFormCitizenship (line 145)
   • ExtractedPosition (line 78) - created in _map_single_item
   • ExtractedBirthplace (line 94) - created in _map_single_item
   • ExtractedCitizenship (line 158) - created in _map_single_item

Considerations

Text Normalization

The source content is normalized (" ".join(text.split())), so the proof might need similar normalization for matching:

def normalize_text(text: str) -> str:
    """Normalize text for proof matching."""
    return " ".join(text.split())

# In validator:
if normalize_text(self.proof) not in normalize_text(content):
    raise ValueError(...)

Substring vs. Exact Match

Should we require:

• Exact substring match (recommended for quotes)
• Fuzzy matching for slight variations
• Token-based overlap (e.g., 80% of words present)

Performance

• Substring search is O(n*m) but fast for reasonable text lengths
• Consider caching normalized content if validating many proofs

Two-Stage Extraction

For two-stage extraction (_map_single_item), the free-form models are validated first, then ExtractedPosition/ExtractedBirthplace/ExtractedCitizenship are created manually. Need to ensure content is available at both stages.

Testing

Add tests to verify:

1. Valid proofs pass validation
2. Invalid proofs are rejected
3. Edge cases (empty proofs, very long proofs, special characters)
4. Normalized text matches work correctly

Related Files

• poliloom/enrichment.py - All extraction logic
• poliloom/models/politician.py - Property model with supporting_quotes field
• tests/test_enrichment.py - Add validation tests

Start in poliloom/enrichment.py by tracing _fetch_and_extract_from_page, extract_properties_generic, extract_two_stage_generic, and _map_single_item. Review the proof-bearing models and tests/test_enrichment.py, then determine how source content should be passed and normalized. Done means valid quotes are retained, invalid proofs are rejected, and the listed edge cases are covered by tests.