sillsdev / sillsdev/interlinearizer-extension
Alignment-based suggestions from a major-language model text
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2
- Forks
- 0
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 55
Description
Split out of #148 (2026-08-12). #148 now covers the surface-form tier — a sibling interlinear project whose approved glosses match the source by surface form — which is ready to build. This issue covers the other kind of model source, which is not.
A major-language Scripture project (e.g. an English Bible) whose words are the gloss vocabulary. It is in a different language from the source, so surface-form matching does not apply: suggestions come from source↔model word alignment — source word → aligned model word(s) → suggested gloss.
Required for full PT9 interlinearization support (Options 1 & 3), which is a stated goal, so this is on the PT9 parity path even though #150's import pipeline does not depend on it.
The decision that gates this
#148 left the approach open, and it has not been answered. It should be settled before any code is written, because the three answers have completely different costs and completely different dependencies.
A. Seed from imported PT9 cluster→gloss data. PT9 already recorded which source word maps to which gloss; reuse that as the alignment. No aligner, no binaries, no licensing question.
Cost: low. But it inverts a dependency — the alignment data comes from #150's import, so this route makes this issue depend on #150, rather than the other way round. Note #150 originally listed #148 as one of its dependencies; that edge was removed on 2026-08-12 because #150's pipeline does not need it. Choosing A would mean adding #218 blocked by #150.
Also: only covers text PT9 already aligned. A project with no PT9 import gets nothing.
B. Compute alignment with fast_align. Apache-2.0, so no license conflict. Still needs per-platform binaries built and shipped, and the createProcess elevated privilege.
C. Compute alignment with eflomal. #102 investigated this in depth and its assessment applies only to this branch. Two blockers it identifies are organizational rather than code, and neither is resolved or separately tracked:
- GPLv3 (eflomal) vs MIT (this extension) — redistribution "needs legal/licensing sign-off from SIL."
- Per-platform binary build/ship infrastructure — win32/linux/darwin with OpenMP, and the stock Makefile's
-march=nativemust be dropped for a portable baseline or the binary may fault on other CPUs.
Options B and C additionally require adding "createProcess" to the currently-empty elevatedPrivileges array in manifest.json — the extension spawns no subprocesses today, so this would be the first.
Scope once the approach is settled
- Resolve
modelProjectId(added by #148) pointing at a non-interlinear Scripture project to the alignment strategy rather than the pool. - The alignment step itself, per the chosen option.
- Feed aligned model words into
deriveTokenSuggestionascandidates, reusing theproducer: "model:{id}"marker #148 introduces soSuggestionDropdownbadges them "from model". - Respect the
interlinearizer.useModelSuggestionssetting from #148.
Persistence is a separate open question
#148 noted alignments "could be persisted via AlignmentLink". Two things to know before choosing that:
projectStorage.updateAnalysiswrites only theanalysisfield, notlinks— there is no persistence path for alignment links at all today.- #129 (Burrito alignment specification) observes that
AlignmentLinkis currently dead weight, so the cost of adopting or mapping to the Burrito flavor "is near zero today and grows with every feature that touches alignment," and recommends doing that investigation before alignment work is scheduled. If this issue persists viaAlignmentLink, #129 should be answered first.
Computing alignment on demand and not persisting it sidesteps both.
Open questions
- Which approach — A, B, or C? Everything else follows.
- Persist via
AlignmentLink, or recompute? See above; if persisting, sequence #129 first. - Alignment granularity and quality. #102 notes an aligner "needs reasonable sentence counts to align well; single-verse alignment in isolation may be weak" — so whole-book or corpus-trained priors may be necessary, which affects whether this is a batch or on-demand operation.
Blocked by: #148 (introduces modelProjectId, the model tier in deriveTokenSuggestion, and the producer marker).
Related: #150, #102, #129, #94.
Size: L
Priority: P2 — prerequisite for full PT9 import parity; not blocking gloss-only use or #150's pipeline.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading #148 and resolving the A/B/C alignment decision, then review deriveTokenSuggestion and the modelProjectId flow it introduces. Inspect manifest.json, projectStorage.updateAnalysis, AlignmentLink, and the interlinearizer.useModelSuggestions setting; done means aligned model words produce model-badged suggestions with an agreed persistence strategy and dependency sequence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100