sillsdev / sillsdev/interlinearizer-extension
Decide the token/segment identity re-anchoring strategy
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2
- Forks
- 0
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 55
Description
Token.ref embeds the verse SID and character offset ("GEN 1:1:0"), and Segment.id is the verse SID. Every analysis link, phrase link, and alignment endpoint joins on these strings. Any change that shifts offsets — an upstream text edit, a tokenizer improvement, or the user actions in #43/#49 — re-keys every subsequent token in the segment and orphans its links. Drift is currently detected (surface-text snapshots flip links to stale) but never healed, and the WebView silently filters dangling links (analysisSlice.ts resolveApprovedAnalysis).
Decide between (at least):
Stable synthetic token ids (UUID or content-position hybrid) with a ref-mapping layer built at tokenization time. Clean identity, but requires persisting the token layer (today it is rebuilt from USJ and never stored).- A documented re-anchoring algorithm that runs on retokenization: match stored snapshots (
surfaceText+ neighborhood) against new tokens within the segment, rewrite refs, and mark unmatched linksstale. Keeps the model as-is; the algorithm becomes the contract. Hybrid: keep derived refs but persist a per-project re-anchor journal of user token operations (#43's splits/joins) that replays after tokenization.
Deliverable: a short design doc in docs/, agreed by the team, plus a spike validating the chosen approach on a real text edit. #43 should not start until this lands.
Size: M (design doc + spike)
Priority: P1 — blocks #43
Updates:
- This is less of an issue than stated because of the analysis suggestion.
- We're avoiding the complexity of FW's attempts at this. We should got with a minimal option 2 to recover from simple shifts/removals/insertions but not worry about complicated edits or significant diffs.
- The target level of effort should be documented with the final code so that future devs/AI don't make it more complicated.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the token and segment identity described here, then inspect analysisSlice.ts, especially resolveApprovedAnalysis, and the existing USJ tokenization flow. Compare the listed strategies with the constraints from #43 and #49. Done means a docs/ design document and a spike on a real text edit, with the minimal re-anchoring scope and its limits documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100