[claude] Detect and recover when OpaqueChanges become applicable
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 14
- Forks
- 4
- Avg merge
- 3d 21h
- Merged PRs (30d)
- 3
Description
[Claude, autonomous]
Problem
Since #80, a change with an unknown $type is stored and synced onward as an OpaqueChange, but SnapshotWorker skips applying it. Snapshots and projected tables are then silently missing its effect, and:
- When the app later registers the type, nothing notices. The stored JSON would now deserialize concretely, but snapshots are never re-folded, so the stale state persists forever.
- Some types will never become known (a different app writing to the same project). That's a legitimate steady state, but a client currently can't even answer "is my materialized state incomplete?", which matters for consumers that write materialized state into other systems (FwHeadless → FieldWorks).
- If the first change to an entity is opaque, no snapshot exists, so later known changes to that entity are skipped too.
Recovery itself is the easy part: the commit log is complete everywhere (opaque JSON round-trips verbatim), so RegenerateSnapshots() already repairs state once the type is registered. What's missing is detection, and ideally a cheaper replay scope than "everything".
One idea: an UnappliedChangeTypes table
One aggregate row per unknown discriminator, written at commit ingest (CrdtRepository.AddCommits, same transaction): TypeName (PK), EarliestCommitId, ChangeCount.
- Detection = intersect
TypeNames with registered discriminators at project open. Non-empty: replay from the predecessor of the earliest affected commit (the existingDeleteStaleSnapshots+UpdateSnapshotspath), then delete the recovered rows. - Never-known types never intersect; their rows sit inert and double as a cheap "this project has changes I can't apply" signal for UI or sync guards.
- Existing DBs backfill with one
json_extract(Change, '$."$type"')scan overChangeEntities.
Ingest is deliberate: the converter also runs on read paths, and the SnapshotWorker skip sites also fire for ordinary out-of-order arrivals, so ingest is the only place that sees each commit exactly once.
Alternatives
Persist nothing; fingerprint the registered type set. On growth, scan history for the new types; regenerate if found.
- Pro: zero bookkeeping, fully retroactive.
- Con: a scan every release that adds a type; "am I incomplete right now?" also costs a scan; the registered set can differ per host (e.g. conditional
AddRemoteResourceEntity), so set comparison needs care.
Per-skip rows (CommitId, ChangeIndex, TypeName, EntityId).
- Pro: exact.
- Con: unbounded in the never-known steady state; no benefit over the aggregate, since replay is scoped by earliest commit, not per change; only records skips after it ships, so it needs the backfill anyway.
Regenerate whenever the registered type set changes.
- Pro: no detection code.
- Con: full replay after nearly every release;
RegenerateSnapshotsis currently unlocked, non-transactional, and slow on big projects, so everyone pays for a rare event.
Do nothing; rely on the manual regenerate troubleshooting action.
- Pro: zero code.
- Con: staleness is silent, so nobody knows to press the button; FwHeadless would sync stale state indefinitely.
Notes
- Whichever way this goes, it should land in the same release as #80 or close behind: builds that skip changes without recording anything leave stale snapshots that later detection can't tell from a clean DB without the backfill scan.
- Auto-regeneration should wait until
RegenerateSnapshotsis locked, transactional, and fast enough. Detection plus a visible warning is already useful on its own.
Context: sillsdev/languageforge-lexbox#2482 (review discussion that prompted this).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading CrdtRepository.AddCommits, SnapshotWorker, and the existing RegenerateSnapshots, DeleteStaleSnapshots, and UpdateSnapshots paths. Compare the proposed aggregate tracking and fingerprint alternatives against the commit-log and ChangeEntities behavior described here. Done should include a decided detection scope, handling for existing databases, and a way to report incomplete materialized state without silently regenerating it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100