evaluation: add a bounded LoCoMo-Plus (cognitive memory) workload

Goal

Add a bounded, reproducible LoCoMo-Plus workload for PowerContext as an evaluation-specific child of #1263 and a companion to the existing LoCoMo path.

This issue captures the benchmark survey and proposes adoption. It is independent evaluation work; its completion is not a prerequisite for closing the core product journeys.

Upstream:

• Paper: Li, Y., Guo, W., Zhang, L., et al. (2026). Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents. ACL 2026. arXiv:2602.10715 · ACL Anthology 2026.acl-long.1150
• Code & data: xjtuleeyf/Locomo-Plus

Why this benchmark (Research basis)

LoCoMo-Plus is a real, independent extension benchmark (Xi'an Jiaotong University + Tencent; ACL 2026), not an alias of LoCoMo. It targets the gap that plain LoCoMo only measures Level-1 explicit factual recall and misses Level-2 cognitive memory: whether an agent preserves and applies the implicit user state / goal / value / causal constraints expressed early in a long conversation.

Core difference vs LoCoMo. On top of LoCoMo's five QA types (single-hop, multi-hop, temporal, commonsense, adversarial), LoCoMo-Plus adds a sixth type Cognitive and a constraint-consistency evaluation framework. Cognitive questions use a cue–trigger semantic disconnect: the trigger query is semantically disconnected from the original clue, so the model must surface a latent constraint it previously absorbed rather than re-read a stated fact. Cognitive memory is decomposed into four latent constraint types: causal / state / goal / value.

Construction pipeline (5 stages). LLM-generated implicit clues → human verification of memory value → trigger-query construction → BM25 + MPNet semantic filtering to remove shortcut paths → human verification of memory elicitation. Cognitive instances are then embedded into LoCoMo long dialogues.

QA distribution. single-hop 840 · multi-hop 290 · temporal 325 · commonsense 100 · adversarial 450 · cognitive 430.

Metrics. Unified dialogue input (task type not exposed to the model) + LLM-as-Judge scoring (correct 1 / partial 0.5 / wrong 0). Human–LLM judge agreement 0.801–0.903.

Key results (author-reported). Factual → cognitive memory drops 30–46 points across models: gemini-2.5-pro 71.78% → 26.06%, gpt-4o 62.99% → 21.05%; dedicated memory systems (A-Mem, Mem0, SeCom) collapse on cognitive too. Cognitive memory degrades sharply with dialogue length — PDR 5.88% vs episodic 67.39%, object 92.85%. The authors also flag two LoCoMo-side biases: a task-disclosure prompt bias and a string-match length bias (metric peaks at the 5.18-token gold-standard length).

Limitations (author-stated). Small scale (diagnostic, not training); English-only; depends on a closed-source LLM judge; no belief revision / emotion / multi-agent coverage.

Community signal. Independent analysis at lhl/agentic-memory endorses the "memory ≠ recall facts" contribution and warns that constraint-type memory should be a separately governed layer to avoid "memory learns policy".

Existing foundations and precise gaps

Foundation already in PowerContext	Gap that LoCoMo-Plus addresses
LoCoMo workload (explicit factual QA over long dialogues)	No Level-2 cognitive / constraint-consistency coverage; cannot measure whether implicit user constraints survive long context
Bounded recall + Source evidence chain	Cognitive PDR collapse (5.88%) shows constraints are dropped over long dialogue; LoCoMo-Plus gives a measurable behavioral-consistency target
Harbor E2E harness (#1263) as shared benchmark basis	Needs a cognitive-type adapter that hides task type (unified input) and scores with an explicit LLM-judge protocol
LLM-as-Judge patterns already used elsewhere	Must pin judge model/version and report human–LLM agreement, mirroring upstream 0.801–0.903

Scope

• Implement a PowerContext memory backend/adapter through supported public interfaces.
• Preserve upstream dataset, question, context-budget, answer, and scoring semantics (unified input + LLM-as-Judge).
• Provide a small fixed smoke subset covering the cognitive constraint types (causal / state / goal / value).
• Provide a separately documented full-run configuration.
• Record exact dataset, harness, model, PowerContext, integration, and policy revisions.
• Keep evaluation artifacts independent from normal Runtime persistence unless explicitly configured.

Required report

Report together:

• answer/task accuracy (overall and per QA type, with cognitive broken out);
• query/retrieval latency;
• prepare latency;
• context bytes/tokens;
• ingestion tokens/cost;
• exact citation availability where applicable;
• abstention behavior;
• infrastructure, retrieval, generation, and judge failures separately.

Any partial run must be labeled subset. A smoke subset must not be presented as a complete benchmark result.

Experiment arms

The initial adapter should support fair comparison of:

• current Memory/Experience granularity;
• query-time compact index;
• write-time L0/L1 summary, only as an experiment;
• hybrid retrieval;
• temporal filtering;
• task-lensed selection.

These arms do not authorize public Memory schema changes. Persistent L0/L1 or temporal fields require separate evidence and an accepted RFC.

Acceptance criteria

• A one-command local smoke run produces complete, inspectable artifacts.
• The adapter uses normal PowerContext interfaces rather than benchmark-only private writes.
• Dataset and harness revisions are pinned.
• Model and judge settings are explicit and reproducible.
• Failures are classified rather than converted to incorrect answers.
• Token/latency/cost accounting is reported with accuracy.
• Subset and full-run reports are visibly distinct.
• A replay of the same saved outputs produces the same deterministic scoring inputs.
• Documentation explains what the benchmark does and does not prove about behavioral consistency vs factual recall.

Non-goals

• Replacing LoCoMo or SWE-bench Pro immediately.
• Claiming benchmark leadership from a partial run.
• Modifying upstream gold labels or prompts to favor PowerContext.
• Treating one benchmark score as product acceptance.
• Adding a general dataset registry or evaluation platform.

Coordination

• Track shared benchmark infrastructure under #1263.
• Reuse baseline work from the existing LoCoMo path where it fits.
• Keep PowerContext-native Work Continuity acceptance in its own Issue.

References

• LoCoMo-Plus paper (arXiv HTML): https://arxiv.org/html/2602.10715v1
• LoCoMo-Plus arXiv abstract: https://arxiv.org/abs/2602.10715
• ACL Anthology 2026.acl-long.1150: https://aclanthology.org/2026.acl-long.1150
• ACL 2026 PDF: https://aclanthology.org/2026.acl-long.1150.pdf
• ar5iv mirror: https://ar5iv.labs.arxiv.org/html/2602.10715
• GitHub xjtuleeyf/Locomo-Plus: https://github.com/xjtuleeyf/Locomo-Plus
• Hugging Face Papers: https://huggingface.co/papers/2602.10715
• Emergent Mind analysis (per-model scores): https://api.emergentmind.com/papers/2602.10715
• alphaXiv overview: https://alphaxiv.org/overview/2602.10715
• Memory Papers summary: https://memorypapers.org/papers/locomo-plus-beyond-factual-cognitive-memory-evaluation-framework-for-llm-agents
• Papers by Luna Dong: https://papers.lunadong.com/paper/5875
• lhl/agentic-memory independent analysis: https://github.com/lhl/agentic-memory/blob/HEAD/ANALYSIS-arxiv-2602.10715-locomoplus.md
• Original LoCoMo paper (Maharana et al., 2024): https://arxiv.org/abs/2402.17753
• Original LoCoMo code snap-research/LoCoMo: https://github.com/snap-research/LoCoMo

Start by reading the Harbor E2E harness and the existing LoCoMo path, then inspect the supported public PowerContext Memory/Experience interfaces. Define the bounded smoke and full-run configurations around the pinned dataset, harness, model, and judge settings. Done means a one-command smoke run produces inspectable artifacts with classified failures, accuracy, latency, token, cost, and subset/full-run reporting.