MemberJunction / MemberJunction/MJ
Cache-invalidation broadcasts are per-row, not per unit of work — a bulk write emits hundreds of client notifications for one logical transaction
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 323
Description
## Problem
The global listener in `packages/MJServer/src/index.ts` publishes a `CACHE_INVALIDATION` message to every connected client on **every** `BaseEntity` save/delete. Nothing coalesces writes that belong to one logical unit of work, so any multi-row operation fans out one broadcast per row:
- `ReconcileFieldPermissions` (field-level security) saves permission rows one at a time inside `RunInEntityTransaction` — enabling field security on `MJ: Employees` writes **84 rows → 84 broadcasts** for what is logically one administrative action.
- A `TransactionGroup` submit (`ExecuteTransactionGroup`) executes its items as individual `BaseEntity.Save()` calls inside one SQL transaction — each raises its own event post-commit, so a future "Permissions dashboard" batching entity/EFP/role edits into one atomic Save All would still emit one broadcast per record.
- The same applies to every server-side bulk write that goes through the entity layer.
Impact is **performance and noise, not correctness**: every broadcast carries accurate data and clients converge on the right state. But each message wakes every connected browser's subscription handler, drives per-message `BaseEngine` processing, and (for filtered engine caches) can trigger repeated invalidate-and-refetch cycles for what should be one refresh.
## What does NOT work (documented dead end — do not re-walk)
Suppressing per-node publishes between `graph_save_started` and `graph_save`. Those events are raised by `BaseEntity`'s ENTITY-GRAPH save path; reconciliation uses `RunInEntityTransaction` with individual `Save()` calls, which is not a graph save — `graph_save` never fires and the suppression is silently inert.
## Design shapes
**(a) An explicit batch scope (preferred direction).** A primitive in MJCore — where `BaseEntity` events originate — marking "a bulk unit of work is in flight"; the MJServer listener coalesces while it is open and publishes once (per entity, or one summary message) on close. Dependency direction constrains placement: MJCoreEntitiesServer cannot call into MJServer, so the primitive cannot live in MJServer. Ideally the scope IS the transaction: `RunInEntityTransaction` and the `TransactionGroup` executor already know the unit of work's boundaries, so hanging the batch on them makes every server-side multi-row write coalesce **by default** rather than only for callers that remember to opt in.
**(b) Debounce the listener per entity name — RULED OUT.** Much simpler, but it drops `RecordData` for coalesced saves, which browser `BaseEngine` caches rely on for in-place updates — coalesced events would degrade to invalidate-and-refetch, breaking important existing behavior. Not an acceptable trade; recorded here only so it isn't re-proposed. Any coalesced message shape must preserve the ability for clients to update their caches in place (e.g. a batch message carrying the per-record payloads, rather than an entity-level message with no row content).
## Related issues — design these together
- #3242 — the same broadcast surface has **no permission filter** on `RecordData`. A coalescing redesign and a filtering redesign should be one design exercise, not two patches: e.g. a coalesced entity-level message with no row content sidesteps both the fan-out volume and the row-content leak for bulk writes.
- #4083 — the opposite failure on the same surface (`mj sync push` publishes no invalidations at all). Whatever contract comes out of this issue should state what a NON-entity-layer bulk write is supposed to publish.
- #2125 — configurable real-time change notifications (cross-tab/user/org scopes); an umbrella if the channel is redesigned.
## Data point from the FLS work
The server's own metadata staleness is already handled correctly under bursts: `ProviderBase`'s dataset-membership refresh debounces 500ms and performs ONE metadata reload per burst (the 84-row flag flip costs one ~2s background refresh). The remaining per-row cost is purely the browser-bound broadcast fan-out — which is why this was accepted as a known gap in the FLS PR rather than blocking it. Original write-up: the FLS branch's `plans/fls-redesign-progress.md`, "KNOWN GAP — one cache invalidation per row, not per transaction" (decided 2026-08-08).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Research direction
Start with the global listener in packages/MJServer/src/index.ts, then trace BaseEntity events through RunInEntityTransaction and the TransactionGroup executor. Define the batch boundary and message contract with issues #3242 and #4083 in view; done means one logical bulk write no longer fans out per-row notifications while preserving required RecordData or an agreed safe replacement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql, typescript
- Domain
- api, backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100