OpenFn / OpenFn/lightning

Collaboration Y.Doc document-state table grows unbounded

Open
#4,930 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug ready
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

What's broken

collaboration_document_states has grown to about 1 GB and keeps growing, almost entirely stale update rows that never get compacted or pruned.

Pruning only runs in-process while a document has an active session, and only deletes update rows once a checkpoint exists for that document. Checkpoints are only created after 500 updates, which most documents never reach, so they never get a checkpoint and never get pruned. Nothing sweeps documents with no active session, so once a room closes its rows sit there indefinitely.

Evidence

  • About 1,062 MB.
  • 5,893 documents, 123,727 update rows, 605 checkpoints.
  • 97% of documents (5,738) have no checkpoint.
  • 99% of update rows (122,968) are older than a day and still present.

Why it matters

It is roughly five times the size of workflow_snapshots and grows with editor usage. Update rows are only needed to reconstruct an open document; once a room closes they can be compacted to a single state or dropped.

What to fix

A background job that compacts and prunes across all documents, not just the ones currently open. Some combination of: compact to a checkpoint on unbind when the last user leaves, a scheduled sweep that checkpoints and prunes documents carrying many stale updates, and a lower checkpoint threshold so compaction actually fires during normal use. Retain enough recent state to reconstruct any document that gets reopened.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the collaboration_document_states lifecycle, checkpoint creation threshold, and in-process pruning described in the issue. Examine what happens when the last user leaves a room and how closed documents could be revisited. Done means stale update rows are compacted or pruned across inactive documents while reopened documents remain reconstructable.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.