PT9 import mishandles parses whose morpheme forms don't sum to the token surface: dropped alone, silently accepted alongside a word gloss

PT9 anchors a cluster to a word with <Range Index="28" Length="4" />. We ignore those offsets — correctly, since they index marker-bearing USFM rather than our baseline text — and re-anchor by matching forms against the token's folded surface (clusterAnchoring.ts:179-190; normalizeSurfaceForm NFC-folds and lowercases only, analysis-identity.ts:35). The range survives as ordering and a proportional prior.

The asymmetry is the bug: PT9 never needs the forms to know which word a parse belongs to, so it accepts any parse the user enters, segmenting or not. We require the forms to sum to the surface — and what happens when they don't depends on whether a word cluster shares the range.

Two failure modes

expected comes from the word cluster's form when the range group has one, and only otherwise from the parse's concatenation (clusterAnchoring.ts:179-181); both facets then ride along to the chosen token (:198-213).

PT9 shape	our behavior	severity
parse cluster alone, forms don't sum to the surface	formMismatch, both facets dropped	data loss, counted
word + parse cluster at one range	anchors on the word's form, parse imported unchecked	wrong morphemes persisted, counted nowhere

The second is the worse of the two and was not obvious: a word gloss acts as a passport for any breakdown that happens to accompany it.

Measured, real files through the real pipeline

A quasi-realistic mini project (hand-built for testing, not field data) run through paranext-core's reader and then convertPt9Project:

clustersTotal=29  converted=25  phrases=3  ambiguous=4
clusterDrops: {verseNotFound:0, formMismatch:1, lemmaOrOther:0, duplicateCluster:0, unparseableLexemeId:0}
tokenAnalyses=26  links=23
analyses accepted whose morpheme forms don't sum to their surface: 4

Of those four, one came through anchoring and is this issue:

surface	morphemes	how
from	["fr", "-om"]	parse cluster at 28-4, anchored by Word:from at the same range, glossed and linked

The other three (thanksgiving ← ["", "giving"], thanksgiving ← ["", ""], prayer ← [""]) are bare payloads from WordAnalyses.xml, which never pass through anchoring at all — they belong to #324, not here.

The single counted drop is not an instance of this bug either: it is ["Stem:", "Stem:giving"] at 78-12, which sits in section-heading text that our extractor excludes from the baseline, so no candidate token existed. It is counted as formMismatch, misattributing the cause — tracked separately as #319.

So on this data the two modes are one drop against one silent acceptance, and the sole formMismatch in the report is really a heading problem wearing this issue's label.

History of the from case, which is the whole argument

Typed as fr -om in PT9's parse editor, it first existed as a parse cluster alone and dropped as formMismatch. After a word gloss was added to the same word, the identical parse imported — glossed desde, morphemes fr + -om, nothing counted. Same PT9 data, opposite outcomes, decided by whether a neighbour cluster happened to anchor the range.

Candidate fixes

• Validate the parse even when a word cluster anchors the group. Closes the silent mode: compare the parse's forms to the chosen token independently of how the group anchored, and count or report a mismatch rather than persisting it unremarked.
• Separator folding (with validation above, this is the whole fix). Fold indicator characters (-, =, ~) out of both sides before comparing, which recovers hand-typed breakdowns like fr + -om and thanks + - + giving. Scope note: PT9 never writes an affix type (its parse editor types every morpheme Stem, and FieldWorks affixes are not matched — see #318 and #130), so folding only ever has to handle indicator characters that users typed into forms. Must share its normalization with #130's indicator rendering.
• Range-derived fallback for parses that still match nothing: rejected, see the stale-range test below. PT9's stored offsets are not maintained across text edits and PT9 itself does not rely on them, so anchoring by range would follow data PT9 has already abandoned.
• Surface the outcome per verse rather than as a bare count (#140).

Relations

• #285 — origin; establishes that blank forms are legal, glossable PT9 data.
• #319 — owns the one formMismatch in the numbers above.
• #130 — shares the normalization question.
• #140 — the reporting half.
• Not #136, which is about re-keying our own token refs across text edits. This is about matching PT9's clusters onto tokens at import time.

Tested: PT9's stored ranges go stale and PT9 does not rely on them

Inserting all before you in PHP 1:2 shifted every offset past index 24 by +4. PT9 did not rewrite the interlinear file - the SFM changed, the cluster data did not - so the stored ranges now point at the wrong text:

stored	points at after the edit
Word:you @24	all
Word:from @28	you
Word:and @48	her, inside "Father"
Phrase:jesus christ @61	ord, inside "Lord"

The analyses still displayed correctly in PT9's Interlinearizer. So PT9 matches an analysis to a word by lexeme form, the same strategy clusterAnchoring uses, and the stored Range is a hint it is willing to leave wrong.

Three conclusions:

1. Form matching is the right ground truth, and clusterAnchoring's doc comment should say so more strongly: the offsets do not merely index a different string, they go stale against their own text and PT9 abandons them.
2. The range fallback is out. Anchoring by a stale offset would place a parse on whatever word the drift lands on.
3. The proportional prior degrades silently. pickByProportionalPrior resolves ambiguity using index / verseExtent against a stale index, so in a project with edit history it can bias toward the wrong candidate among equal-folding tokens - the more editing since the analyses were made, the worse. Worth deciding whether to weaken it toward pure order-based matching, or to keep it and document the limit.

The prior is skewed before any staleness enters

A second distortion, independent of the one above: PT9's indices are measured against its own string for the verse, which carries the verse marker. In PHP 1:2 of the same project, every PT9 index is the verse-text position plus 5, the width of  2 :

word	index in verse text	PT9 Range Index
Grace	0	5
peace	10	15
you	19	24
from	23	28
Jesus Christ	56	61

pickByProportionalPrior compares clusterIndex / verseExtent against wordTokens[candidate].charStart / baselineLength, so the two ratios come from differently-origined scales. The offset is constant per verse and therefore matters most in short verses, and it is present in freshly written data - staleness only adds to it. Whatever is decided for conclusion 3, the prefix should be accounted for or the prior dropped, not left implicit.

Size: M — matcher change plus fixtures.
Priority: P1 — one mode loses user data, the other persists unvalidated morphemes with no counter.

Start in clusterAnchoring.ts:179-213 and compare its group-level anchoring with normalizeSurfaceForm in analysis-identity.ts:35, then trace the path through convertPt9Project. Add coverage for a parse sharing a range with a word cluster and for separator folding, with completion defined by mismatches being validated and reported rather than silently persisted; also verify the stale-range and proportional-prior behavior described here.