sillsdev / sillsdev/interlinearizer-extension

Dedupe phrase analysis payloads on write

Open
#187 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
2
Forks
0
Avg merge
1d 17h
Merged PRs (30d)
55

Description

createPhrase pushes a fresh UUID payload plus a link every single time. Group ne … pas and gloss it not in fifty places and you get fifty separate PhraseAnalysis records with identical content. Token analyses do the opposite — dedupe on write, one payload shared across all usages.

Three consequences, and they are bigger than storage bloat:

  • Phrases get no suggestions at all. buildPoolIndex reads tokenAnalyses only, so there is no phrase pool. Glossing your fifty-first ne … pas inherits nothing; the user retypes it. "Gloss once, applied everywhere" simply does not exist for phrases.
  • Any phrase list is a grouping, not a record set. Editing one entry means a fifty-way fan-out write instead of a single mutation. This is the blocker for phrase rows in the Analysis Catalog (#186).
  • Edit semantics differ silently by type. Editing a token analysis changes all usages because the payload is shared. Editing a phrase would change all usages only because we deliberately fanned out. Same visible behavior, entirely different mechanism.

What to do

Make createPhrase find-or-create against a content-identical PhraseAnalysis (surface form + gloss + senseRef), so one payload maps to many links, exactly as the token path works.

The reducer change is small; the surrounding work is not:

  • Fork-on-edit for phrases — the equivalent of isPayloadSharedByOtherLinks / forkSharedAnalysis, so a per-occurrence edit does not rewrite co-linked phrases.
  • Re-converge on edit — the equivalent of mergeIntoIdenticalPayload, so an edit cannot leave a duplicate the find-or-create path would have avoided.
  • Payload cleanup — drop the payload only when its last link is removed, so removing one phrase never orphans another's analysis.

Migration

Existing drafts already hold many duplicate phrase payloads. Converging them on load is a data migration and wants a schema version first (#137). Decide explicitly whether to converge on load, converge lazily on next write, or leave existing duplicates alone.

Behavior change

Editing one occurrence's phrase gloss currently affects only that occurrence. After dedupe it affects all of them unless forked. That is the same local/global intent problem already worked through for tokens, and it needs the same separation — per-occurrence edits stay keyed by the link, global edits keyed by the analysis id.

Related: #186, #137. See also user-questions.md, "separating per-token edits from global analysis edits".

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at createPhrase and compare its behavior with the token analysis path, including isPayloadSharedByOtherLinks, forkSharedAnalysis, and mergeIntoIdenticalPayload. Read user-questions.md and the related issues #186 and #137 before deciding how migration is handled. Done means deduplicated phrase payloads, safe fork-on-edit and re-convergence, correct last-link cleanup, and an explicit treatment of existing duplicates.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.