MemberJunction / MemberJunction/MJ

Materialized RunQuery: cache the per-read MR status lookup + memoize the order-by parse

Open
#3,794 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
2d 1h
Merged PRs (30d)
323

Description

Follow-up from the Query & Entity Materialization PR (#3735), split out so it survives PR-body deletion on merge.

**Observation (rkihm-BC re-review):** every materialized `RunQuery(DataSource:'Materialized')` read currently pays three uncached metadata operations before a single snapshot row is read:
- `tryBuildMaterializedQueryPlan` does **two** `RunView`s — the `MJ: Materialized Result Queries` join lookup, then the `MJ: Materialized Results` row read with `BypassCache: true`.
- plus an un-memoized `SQLParser.Astify` (`queryHasTopLevelOrderBy`) of the platform SQL.

For a feature whose purpose is making large reads cheap, three uncached metadata ops per read is a real tax, and on small/medium result sets it could be a net loss.

**Constraint:** the `BypassCache` on the status read is deliberate and correct — `DriftHold`/`Disabled` are written out of band by CodeGen via direct SQL (no BaseEntity cache-invalidation event), so a cached `Active` status would keep serving a held/disabled snapshot (the exact stale-serve the DriftHold mechanism prevents). Any cache here must not defeat that.

**Proposed fix:**
- A short-TTL cache keyed on MaterializedResult ID that **any status write invalidates**, so the freshness guarantee is preserved while eliminating the repeated point lookup.
- Memoize `queryHasTopLevelOrderBy` per (query SQL + platform).

Deferred from the PR intentionally (correctness-sensitive vs. the DriftHold freshness guarantee); should be its own focused change.

Contributor guide

Open the contributing guide

Research direction

Start at tryBuildMaterializedQueryPlan and the queryHasTopLevelOrderBy path described in the issue, then review PR #3735 for surrounding materialized-read behavior. Verify the MaterializedResult status lookup and query SQL/platform parse paths, including the DriftHold and Disabled freshness constraint. Done means repeated materialized reads avoid the redundant metadata and parse work without serving an out-of-date status.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.