MemberJunction / MemberJunction/MJ

Cache-invalidation broadcasts are per-row, not per unit of work — a bulk write emits hundreds of client notifications for one logical transaction

Open
#4,250 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
2d 1h
Merged PRs (30d)
323

Description

## Problem

The global listener in `packages/MJServer/src/index.ts` publishes a `CACHE_INVALIDATION` message to every connected client on **every** `BaseEntity` save/delete. Nothing coalesces writes that belong to one logical unit of work, so any multi-row operation fans out one broadcast per row:

- `ReconcileFieldPermissions` (field-level security) saves permission rows one at a time inside `RunInEntityTransaction` — enabling field security on `MJ: Employees` writes **84 rows → 84 broadcasts** for what is logically one administrative action.
- A `TransactionGroup` submit (`ExecuteTransactionGroup`) executes its items as individual `BaseEntity.Save()` calls inside one SQL transaction — each raises its own event post-commit, so a future "Permissions dashboard" batching entity/EFP/role edits into one atomic Save All would still emit one broadcast per record.
- The same applies to every server-side bulk write that goes through the entity layer.

Impact is **performance and noise, not correctness**: every broadcast carries accurate data and clients converge on the right state. But each message wakes every connected browser's subscription handler, drives per-message `BaseEngine` processing, and (for filtered engine caches) can trigger repeated invalidate-and-refetch cycles for what should be one refresh.

## What does NOT work (documented dead end — do not re-walk)

Suppressing per-node publishes between `graph_save_started` and `graph_save`. Those events are raised by `BaseEntity`'s ENTITY-GRAPH save path; reconciliation uses `RunInEntityTransaction` with individual `Save()` calls, which is not a graph save — `graph_save` never fires and the suppression is silently inert.

## Design shapes

**(a) An explicit batch scope (preferred direction).** A primitive in MJCore — where `BaseEntity` events originate — marking "a bulk unit of work is in flight"; the MJServer listener coalesces while it is open and publishes once (per entity, or one summary message) on close. Dependency direction constrains placement: MJCoreEntitiesServer cannot call into MJServer, so the primitive cannot live in MJServer. Ideally the scope IS the transaction: `RunInEntityTransaction` and the `TransactionGroup` executor already know the unit of work's boundaries, so hanging the batch on them makes every server-side multi-row write coalesce **by default** rather than only for callers that remember to opt in.

**(b) Debounce the listener per entity name — RULED OUT.** Much simpler, but it drops `RecordData` for coalesced saves, which browser `BaseEngine` caches rely on for in-place updates — coalesced events would degrade to invalidate-and-refetch, breaking important existing behavior. Not an acceptable trade; recorded here only so it isn't re-proposed. Any coalesced message shape must preserve the ability for clients to update their caches in place (e.g. a batch message carrying the per-record payloads, rather than an entity-level message with no row content).

## Related issues — design these together

- #3242 — the same broadcast surface has **no permission filter** on `RecordData`. A coalescing redesign and a filtering redesign should be one design exercise, not two patches: e.g. a coalesced entity-level message with no row content sidesteps both the fan-out volume and the row-content leak for bulk writes.
- #4083 — the opposite failure on the same surface (`mj sync push` publishes no invalidations at all). Whatever contract comes out of this issue should state what a NON-entity-layer bulk write is supposed to publish.
- #2125 — configurable real-time change notifications (cross-tab/user/org scopes); an umbrella if the channel is redesigned.

## Data point from the FLS work

The server's own metadata staleness is already handled correctly under bursts: `ProviderBase`'s dataset-membership refresh debounces 500ms and performs ONE metadata reload per burst (the 84-row flag flip costs one ~2s background refresh). The remaining per-row cost is purely the browser-bound broadcast fan-out — which is why this was accepted as a known gap in the FLS PR rather than blocking it. Original write-up: the FLS branch's `plans/fls-redesign-progress.md`, "KNOWN GAP — one cache invalidation per row, not per transaction" (decided 2026-08-08).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start with the global listener in packages/MJServer/src/index.ts, then trace BaseEntity events through RunInEntityTransaction and the TransactionGroup executor. Define the batch boundary and message contract with issues #3242 and #4083 in view; done means one logical bulk write no longer fans out per-row notifications while preserving required RecordData or an agreed safe replacement.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql, typescript
Domain
api, backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.