sillsdev / sillsdev/interlinearizer-extension

FieldWorks enriches word glosses but never morpheme glosses, and imported entries lose their FieldWorks provenance

Open
#318 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2
Forks
0
Avg merge
1d 17h
Merged PRs (30d)
55

Description

Scoped down after testing: a FieldWorks-linked project does import, and ids stay Type:Form-shaped. What is lost is morpheme-level gloss data that FieldWorks holds, and any record of where the mirrored data came from.

Established (Paratext 9.5.110.9, ParatextData 9.5.0.24)

  • The Interlinearizer uses the link. Paratext.Interlinear.InterlinearForm.LoadInterlinearizer calls IntegratedLexicalProvider.GetLexicon(project, allowFlexData: true) and ParatextLexicalAnalyser.Get(project, true); the model text gets false. No Paratext plugin is required as of PT9 9.4 — the integration is built in, so an empty Paratext 9\plugins folder means nothing.

  • The merge is additive. LexiconWithFlexAdditions holds both paratextLexicon and flexLexicon, and EncodesKeyInId delegates to the Paratext half.

  • Word-level selections are mirrored into the file. A FieldWorks entry for "from" produced a plain PT9 entry with a PT9-minted sense id and the FieldWorks gloss copied in:

    <Lexeme Type="Word" Form="from" Homograph="1" />
    <Entry><Sense Id="WYOx2yJ0"><Gloss Language="es">desda</Gloss></Sense></Entry>
    

    referenced as <Lexeme Id="Word:from" GlossId="WYOx2yJ0" />. So parseLexemeKeyId handles it, our resolver finds entry and sense, and the gloss imports. The "opaque FieldWorks ids break every cluster" risk is not real.

  • Morpheme-level lookups never match FieldWorks. In the same project I created a FLEx suffix -ants (gloss agente) and stem serv (gloss servir), restarted PT9, and analyzed "servants" as serv ants. PT9 wrote both morphemes as Type="Stem" with no GlossId at all:

    <Cluster><Range Index="23" Length="8" /><Lexeme Id="Stem:serv" /><Lexeme Id="Stem:ants" /></Cluster>
    

    FieldWorks stores from and serv with the identical morph-type guid, so its data does not distinguish them. The difference is the lookup: lexeme identity is Type + Form + Homograph, the Interlinearizer resolves via the merged lexicon's key indexer (LexiconV2.get_Item, from InterlinearLexemeClusterCreator.Create and the sense guessers), and FieldWorks entries appear to be exposed as Word-typed — so Word:from matched while Stem:serv did not. Inferred from the asymmetry, not traced through the provider's type mapping; a PT9 restart ruled out stale caching.

What is wrong for us

  1. Imported morphemes are unglossed even when FieldWorks has their glosses. Nothing in the files carries servir or agente, so no import can recover them. The user sees blank morpheme glosses in a project whose lexicon is full. We cannot fix this in PT9, but we can explain it, and a future FW Lite / Lexicon-extension lookup could fill it.
  2. No provenance. A mirrored entry is indistinguishable from a hand-authored one; nothing records that its authority is a FieldWorks entry. #227/#228 refs built from imported data will point at PT9's copy, and #229 cannot tell that an entry already exists upstream.
  3. hasAssociatedLexicalProject is ignored. It appears in this repo only as a test-fixture default (convertPt9Project.test.ts:19). At minimum it should tell the user the lexicon has an external source of truth — which is exactly the explanation for (1).
  4. Unused FieldWorks inventory is invisible — entries never used in the Interlinearizer never reach the files. Expected, but it bounds what an import can offer for suggestions.

Open question

Are FieldWorks-only word analyses mirrored? WordAnalysesWithFlexAdditions merges them for PT9's own use. WordAnalyses.xml gained nothing from the tests above, but neither test created a FLEx-side analysis, so this is untested rather than answered.

Size: S for (3), M for (2) — it depends on how refs carry provider identity per #225.
Priority: P2 — nothing is corrupted; it bounds the lexicon-ref work and explains a confusing gap to users.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with convertPt9Project.test.ts at line 19, then trace the provider-identity work referenced by #225 and how imported entries and refs are represented. Define and test the handling of hasAssociatedLexicalProject and FieldWorks provenance, while documenting the unresolved morpheme and word-analysis cases; done when the chosen behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.