Are third-party memory systems in scope, and what is the adapter contract?

Hi, and thanks for maintaining this. It is the closest thing the field has to a
neutral, runnable comparison, which is exactly why I would rather ask than guess.

The question: are third-party memory systems in scope for this suite, and if
so what is the contribution path? The README documents the Mem0 Cloud and Mem0
OSS backends, and I could not find a CONTRIBUTING.md or a documented adapter
interface for anything else, so I do not want to assume either way.

Three sub-questions, in the order they would matter:

1. Scope. Is a third-party system something you would want in this
   repository at all, or is the suite deliberately scoped to Mem0 and intended
   to be forked by others?
2. The contract. If third-party systems are in scope, what is the interface?
   benchmarks/locomo/run.py drives mem0.add(...) and mem0.search(...) from
   benchmarks/common/mem0_client.py, with the answerer and judge separate on
   OPENAI_API_KEY, so that client looks like the natural seam. It is
   Mem0-shaped rather than a declared interface, though, and I would rather
   build to a contract you state than to one I inferred from a call site.
   Is --backend the extension point you have in mind, or would you want
   something else?
3. Results. If an adapter were accepted, would a run under your protocol be
   reportable in results/, and under what conditions? A number produced by
   your harness is worth considerably more than one produced by mine, because it
   takes me out of the scoring loop entirely.

What I would be proposing to add. Engraphy, a self-hosted MCP memory server:
a typed graph on Postgres and pgvector, de-duplication banded before the row
exists, superseded facts retired rather than deleted, and tenants separated by
Postgres row-level security. Source-available under BSL 1.1, converting to
Apache 2.0 in 2030. Embedding runs in-process, so it needs no model provider of
its own; under your harness it would use your answerer and judge like any other
backend.

• Repository: https://github.com/devon-clarkk/engraphy
• Site: https://engraphy.tech

Leading with method rather than a score, on purpose. I have a LoCoMo number
from my own harness, and I am aware that a self-reported LoCoMo percentage
persuades nobody who is paying attention, least of all in this repository, where
issue #10 is already asking the right kind of question about the judge template.
So what I would rather put forward is the method, which is public and checkable:

• The adapter handles the published category mapping explicitly, including the
  adversarial category as an abstention case with its own gold key, rather than
  folding it into the headline.
• It does not flatten speaker names to user and assistant.
• The dataset is pinned by sha256 with the verified counts recorded: 1,986
  questions in locomo10.json, of which 446 are adversarial, so the 1,540
  usually quoted is the file minus that category. Runs report per-category, so
  either denominator can be reconstructed.
• Every run writes a manifest carrying the dataset digest, the model id per
  role, prompt hashes, dedup thresholds read back from the live database, and
  the per-question rows.
• I have just made the whole run reproducible from a clean checkout by anyone
  with an OpenAI-compatible base URL and API key, with no vendor CLI or
  subscription involved. The walkthrough is bench/RUN-LOCOMO.md, in review at
  https://github.com/devon-clarkk/engraphy/pull/8, and it is explicit about which
  parts of my published figure a clean checkout reproduces today and which it
  does not yet.

I have deliberately not opened a pull request against your code. Building an
adapter against an interface you have not documented, then asking you to review
it, seemed like the wrong order. Happy to do the work in whatever shape you
prefer, or to close this if the answer to question 1 is no.

Read benchmarks/locomo/run.py and benchmarks/common/mem0_client.py first, then review the README's Mem0 backend documentation. A completed issue would state whether third-party systems are in scope, define the supported adapter contract and --backend behavior, and explain when adapter results may be included in results/.