mem0ai / mem0ai/memory-benchmarks

Add LongMemEval temporal retrieval diagnostics

Open
#13 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
112
Forks
46
PR merge metrics
No merged PRs in 30d

Description

Following up from mem0ai/mem0#5614, where memory-quality signals were redirected from SDK search() into this benchmark suite.

I'd like to contribute a small LongMemEval-first diagnostic for temporal retrieval validity. LongMemEval seems like the cleanest starting point because each question has a question_date, while retrieved memories may include created_at / updated_at.

Problem

Today LongMemEval reports answer/retrieval quality by cutoff, but it does not expose whether retrieved memories are temporally valid relative to the question date.

That matters because benchmark ingestion can include haystack sessions on both sides of the question timestamp. A retrieved memory may be semantically relevant but dated after the question, which makes it invalid evidence for that question.

Current shape

compute_longmemeval_metrics() currently reports cutoff-level quality like:

metrics_by_cutoff[label] = {
    "overall": {
        "total": total,
        "correct": correct,
        "accuracy": correct / total * 100 if total else 0.0,
        "avg_score": statistics.mean(scores) * 100 if scores else 0.0,
    },
    "by_question_type": type_metrics,
}

Per-question outputs already carry the fields needed for temporal diagnostics:

{
    "question_date": "2023/04/10 (Mon) 10:15",
    "retrieval": {
        "search_results": [
            {
                "memory": "...",
                "score": 0.82,
                "created_at": "2023-04-10T21:32:00Z",
                "updated_at": "2023-04-10T21:32:00Z",
            }
        ]
    },
}

Possible output shape

A first pass could add a deterministic diagnostic block per cutoff:

metrics_by_cutoff[label] = {
    "overall": {...},
    "by_question_type": type_metrics,
    "temporal_retrieval": {
        "timestamp_coverage": 0.94,
        "future_retrieval_rate": 0.02,
        "avg_memory_age_days": 18.7,
    },
}

Where:

  • timestamp_coverage = fraction of retrieved memories with a parseable timestamp;
  • future_retrieval_rate = fraction of retrieved memories dated after question_date;
  • avg_memory_age_days = average age of retrieved memories relative to question_date, when timestamps are available.

I would avoid calling this semantic "staleness" in the first version. This is only timestamp-based retrieval validity, not proof that an old memory is obsolete.

Proposed first scope

  • LongMemEval only.
  • Pure benchmark-side metric computation.
  • No changes to Mem0 search behavior.
  • No LLM judge for temporal validity.
  • Handle missing/invalid timestamps without failing the benchmark run.

This seems complementary to PR #12: that PR adds deterministic failure-mode scenarios, while this would add aggregate temporal retrieval diagnostics to LongMemEval benchmark outputs.

Open question

Would maintainers prefer this to start as:

  1. a shared helper in benchmarks/common/metrics.py, wired first into LongMemEval, or
  2. a LongMemEval-local helper first, generalized later if useful?

If this direction sounds right, I can pick up the first PR.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with compute_longmemeval_metrics() and the existing per-question retrieval outputs, then inspect whether benchmarks/common/metrics.py or the LongMemEval implementation is the better home. Add the proposed temporal_retrieval block for each cutoff, and verify that missing or invalid timestamps do not fail the benchmark run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.