sillsdev / sillsdev/interlinearizer-extension

Decide the token/segment identity re-anchoring strategy

Open
#136 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2
Forks
0
Avg merge
1d 17h
Merged PRs (30d)
55

Description

Token.ref embeds the verse SID and character offset ("GEN 1:1:0"), and Segment.id is the verse SID. Every analysis link, phrase link, and alignment endpoint joins on these strings. Any change that shifts offsets — an upstream text edit, a tokenizer improvement, or the user actions in #43/#49 — re-keys every subsequent token in the segment and orphans its links. Drift is currently detected (surface-text snapshots flip links to stale) but never healed, and the WebView silently filters dangling links (analysisSlice.ts resolveApprovedAnalysis).

Decide between (at least):

  1. Stable synthetic token ids (UUID or content-position hybrid) with a ref-mapping layer built at tokenization time. Clean identity, but requires persisting the token layer (today it is rebuilt from USJ and never stored).
  2. A documented re-anchoring algorithm that runs on retokenization: match stored snapshots (surfaceText + neighborhood) against new tokens within the segment, rewrite refs, and mark unmatched links stale. Keeps the model as-is; the algorithm becomes the contract.
  3. Hybrid: keep derived refs but persist a per-project re-anchor journal of user token operations (#43's splits/joins) that replays after tokenization.

Deliverable: a short design doc in docs/, agreed by the team, plus a spike validating the chosen approach on a real text edit. #43 should not start until this lands.

Size: M (design doc + spike)

Priority: P1 — blocks #43

Updates:

  • This is less of an issue than stated because of the analysis suggestion.
  • We're avoiding the complexity of FW's attempts at this. We should got with a minimal option 2 to recover from simple shifts/removals/insertions but not worry about complicated edits or significant diffs.
  • The target level of effort should be documented with the final code so that future devs/AI don't make it more complicated.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the token and segment identity described here, then inspect analysisSlice.ts, especially resolveApprovedAnalysis, and the existing USJ tokenization flow. Compare the listed strategies with the constraints from #43 and #49. Done means a docs/ design document and a spike on a real text edit, with the minimal re-anchoring scope and its limits documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.