MemberJunction / MemberJunction/MJ
Live tier MG1/MG2: phase-P predicate counts memory writes but never checks the instructed types, so model non-compliance detonates as a product failure
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 323
Description
A test-side defect found during v5.51.0 release triage (DEPLOYMENT.md Step 4). **MJ's behaviour here is correct** — the check misattributes model non-compliance as a product failure.
## Symptom
```
✗ agent-memory-guards.MG1: the allowed 'Preference' write did not land
(got ["rejected-type","rejected-type","rejected-type","rejected-type","rejected-type",
"rejected-type","rejected-type","rejected-type","rejected-type","rejected-type"])
✗ agent-memory-guards.MG2: per-run cap must land exactly 5 written (got 1) — expected 5, got 1
```
## What actually happened
MG1 instructs the model to emit **exactly 2** memory writes — one `Constraint` (disallowed, expected `rejected-type`) and one `Preference` (allowed, expected `written`). The model emitted **10**, all with types that failed validation.
MJ rejected all 10 **correctly**. The allowed types are hardcoded in code, not metadata (`MemoryWriteManager.ts:215`):
```ts
if (request.type !== 'Preference' && request.type !== 'Context') {
return `Memory type "${request.type}" is not allowed in-flight — only 'Preference' and 'Context'. ...`;
}
```
So a seeding/metadata gap is ruled out — there is no data dependency to get wrong.
## The defect
`agent-memory-guards.checks.ts` gates phase-A on a predicate that only counts:
```ts
assertP(dispositions.length >= 2, `IT: Memory Writer emitted ${dispositions.length} writes, expected 2`);
```
It never verifies the model copied the instructed **types** verbatim, which is the whole premise of the seeded `IT: Memory Writer` agent — the bundle header states its prompt *"emits EXACTLY the memoryWrites listed in the user message — same count/order/type/content/scope, verbatim … so every guard is provably ATTEMPTED (anti-vacuity)."*
Because 10 >= 2, phase P passes, and the run falls through to phase-A assertions that can never hold. The result is a hard failure that reads like a product defect (*"the allowed 'Preference' write did not land"*) when the true cause is that the model never emitted a valid `Preference` write.
MG2 fails the same way: it asks for 6 valid `Preference` writes to exercise the per-run cap of 5 and got 1 written.
## Consequence
This is the one failure class DEPLOYMENT.md 4.6 triage cannot classify correctly. `model-noncompliance:` is documented as accepted variance; anything else is *"a real product defect"* and blocks a release. This defect routes variance into the blocking bucket, so a build engineer either wastes time chasing a non-bug or learns to distrust the tier.
## Suggested fix
Make the phase-P predicate assert the dispositions correspond to the instructed writes — matching count **and** types — so a non-compliant model produces `model-noncompliance:` (retried, then classified as variance) rather than a phase-A hard failure. MG3 in the same bundle already does something closer to this and correctly reported `model-noncompliance: MG3 — expected both scoped notes to persist (found 1) (after 3 attempts)`.
## Reproduction context
Reproduced across two full live-suite runs, the second with `AGENT_LIVE_SETTLE_MS=20000 AGENT_SETTLE_MS=20000` on an idle machine — so not timing. The live tier's model ladder landed most work on nano/flash-lite tiers, where multi-step verbatim instruction-following is unreliable by nature; the check needs to tolerate that rather than treat it as a product regression.
Related: #3390, #3391 (same triage).
Contributor guide
Research direction
Start in agent-memory-guards.checks.ts at the phase-P predicate, then read the instructed writes in the MG1 and MG2 checks and the validation rule at MemoryWriteManager.ts:215. Run the live agent-memory-guards suite and confirm that mismatched write types are reported as model-noncompliance rather than reaching phase-A product assertions; compare the existing MG3 behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100