Rewrite quality checks allow grammatically malformed generalizations

Priority Level

Medium (Annoying but has workaround)

Describe the bug

Summary

The rewrite pipeline can produce grammatically malformed sentences when generalizing an entity embedded in a noun phrase. These outputs can still pass the automated quality gate with relatively high utility scores and needs_human_review=False.

Observed behavior

While executing docs/notebook_source/04_rewriting_biographies.py, the pipeline produced:

Luca Hawthorne is a in his late 30s astronomer...

Robert Harper is a in his late 60s paramedic...

The age values were generalized, but the surrounding sentences were not restructured. Grammatically correct versions would be:

Luca Hawthorne is an astronomer in his late 30s...

Robert Harper is a paramedic in his late 60s...

The malformed outputs received the following metrics:

Output	Utility	Leakage	Needs review
Luca Hawthorne…	0.908333	0.0	False
Robert Harper…	0.87	0.0	False

No repair was triggered.

Expected behavior

Generalized values should be integrated into grammatically correct prose. Clearly malformed rewrites should either be repaired automatically or marked for human review.

Likely failure mode

The model appears to substitute the generalized age phrase without restructuring the enclosing noun phrase:

• a 37-year-old astronomer
• a in his late 30s astronomer

Although the rewrite prompt requires natural prose, the current quality evaluation does not reliably detect this grammatical failure.

Impact

• Generated text can be visibly ungrammatical despite passing evaluation.
• Utility scores may overstate output quality.
• needs_human_review does not identify malformed prose because it is score and threshold based.
• Users may consume or publish defective rewrites without an obvious failure signal.

Suggested next steps

1. Add prompt examples showing how attributive phrases must be restructured during generalization.
2. Add explicit grammaticality or fluency criteria to the quality evaluator.
3. Trigger repair when malformed syntax is detected.
4. Study if this is the only case where it is happening, perhaps it's slipping through in other benchmark datasets too.

Steps/Code to reproduce bug

The malformed outputs were generated and committed in:

• 2c005e0b — docs: regenerate tutorial notebooks with OpenRouter

They were identified during review of PR #274:

• Malformed astronomer phrase
• Malformed paramedic phrase

To reproduce:

1. Run docs/notebook_source/04_rewriting_biographies.py using the default rewrite configuration.
2. Inspect result.dataframe["biography_rewritten"].
3. Because generation is nondeterministic, repeated execution may be required.
4. Observe that malformed output can receive a passing utility score and needs_human_review=False.

Expected behavior

Rewrite should generate correct grammar

Additional context

No response

Start by running docs/notebook_source/04_rewriting_biographies.py with the default rewrite configuration and inspect result.dataframe["biography_rewritten"]. Review the malformed examples and the quality metrics showing needs_human_review=False, then determine whether evaluation or repair should catch them. Done means malformed rewrites are corrected or marked for human review, with coverage beyond the reported biography cases.