Null bytes in LLM-extracted supporting quotes crash PostgreSQL insert

Problem

Enrichment fails with invalid byte sequence for encoding "UTF8": 0x00 when storing extracted properties with supporting quotes containing null bytes.

Observed in production on the Papiamento Wikipedia article for Mauritsz de Kort (pap.wikipedia.org).

Error

poliloom.enrichment - ERROR - Error storing extracted data: (pg8000.dbapi.ProgrammingError)
'invalid byte sequence for encoding "UTF8": 0x00'

Root Cause

The supporting_quotes returned by the LLM contain null bytes where the first byte of multi-byte UTF-8 sequences should be. For example:

The leading \xc3 byte of each 2-byte UTF-8 sequence is replaced with \x00. PostgreSQL TEXT columns reject null bytes, causing the insert to fail and rolling back the entire transaction for that page.

Questions to Investigate

• Is this corruption happening in the LLM response, or during text extraction from the HTML (BeautifulSoup.get_text() → whitespace join)?
• Is pg8000 (our DB driver) stricter about this than psycopg2? Could the driver choice matter?
• Should we sanitize at the storage layer (strip null bytes), or earlier in the pipeline (fix encoding before sending to LLM)?
• How common is this across non-Latin Wikipedia editions?

Start by reproducing the Papiamento case and tracing supporting_quotes from BeautifulSoup.get_text() through the enrichment storage path to the pg8000 insert. Compare the received LLM response with the extracted HTML text and investigate whether pg8000 and psycopg2 handle the bytes differently. Done means the corruption source is identified, a pipeline boundary for handling it is chosen, and the PostgreSQL insert no longer rolls back.