afx-team / afx-team/hebb-mind

eval: complete the PersonaMem benchmark report (commit in-tree run-N, run full 589-question dataset)

Aberta
#28 0 comentários 0 reações 0 responsáveis Ver no GitHub
effort: hard eval
Linguagem predominante
Python
Estrelas
52
Forks
18
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

## Summary

PersonaMem is wired into the harness but **has no in-tree benchmark report**, and the published page cites a report file that does not exist. We need to run PersonaMem on the **full dataset** under the house rules, commit the `run-N` report in-tree, and update the public pages so the cited number actually resolves.

## Current state

- **Adapter exists**: `eval/datasets/personamem.py` (loader) + `eval/benchmarks/personamem_bench.py` (runner). `eval/README.md` lists it as `personamem` → *end-to-end QA judge* (no clean retrieval id; preference rewrite).
- **Public page exists**: `repo_pages/benchmarks/personamem/index.md` (+ zh mirror) reports **67.6% QA acc**, sourced to `eval/reports/personamem/v1/run-1/personamem.md`.
- **The cited report is missing** — there is no `eval/reports/personamem/` directory in-tree. The published number is therefore uncited and unreproducible, which violates the house rule: *"A published number in `repo_pages/benchmarks/` MUST cite an in-tree `run-N` report."*
- The 67.6% is also a **partial, stale** measurement: only **37 questions across 3 scenarios** (the full set is **589 questions / 20 personas / 7 question types**, 32k context version), on **v0.1.1** with a **Kimi-K2.5** judge. Per-category it was strongest on `track_full_preference_evolution` (88.9%) and weakest on `recall_user_shared_facts` (40.0%).
- Competitor comparison rows (mem0 / Letta / MemPalace / Zep) are all `TBD`.

## Task — supplement the PersonaMem eval report

1. **Run the full dataset (589 questions, 20 personas, all 7 question types)** on the **current shipped default** config — not the 37-question slice.
2. **House-rule compliant run**:
- Isolated server (own port in `8401–8409`, own workdir `hebb.db`) — never the daily `hebb service` on 8321.
- `.venv/bin/python` (the eval entrypoints need it).
- No train/test split — full dataset (Hebb Mind trains no model).
- Run `mode=raw` (no consolidation) to match the page's "isolate the retrieval layer" framing; optionally also `mode=consolidated` for contrast.
- Report **rerank ON** (shipped default) and ideally **rerank OFF** for the delta.
3. **Metric**: end-to-end QA judge per the README. Document the judge model and reader prompt; prefer an **official / neutral** judge prompt over a benchmark-tuned one (consistent with how LongMemEval QA uses the official judge verbatim). Multiple-choice ground truth (`correct_answer` + `all_options`) is available, so report accuracy with a clear answer-matching rule.
4. **Commit the in-tree report** under `eval/reports/personamem//run-N/personamem.md` so the public citation resolves. Include the **per-category breakdown** across all 7 question types.
5. **Update the published pages together**: EN `repo_pages/benchmarks/personamem/index.md` **and** its `zh/` mirror, replacing the 37q number with the full-dataset `run-N` number + per-category table, and fixing the dangling source link.
6. Fill competitor rows where a public number exists; otherwise leave `TBD` with a note.

## Acceptance criteria

- [ ] `eval/reports/personamem/.../run-N/personamem.md` exists in-tree with the full-589-question result + per-category breakdown.
- [ ] EN + zh benchmark pages cite that in-tree `run-N` report (no dangling citation) and agree on the number.
- [ ] Judge model + reader/judge prompt and run config (mode, rerank, port, version) are recorded in the report.
- [ ] If `mode=raw` and `mode=consolidated` both run, both numbers are reported.

## References

- Dataset: https://github.com/bowen-upenn/PersonaMem · HF `bowen-upenn/PersonaMem`
- `eval/README.md` — datasets/metrics/report layout and invariants.
- `eval/datasets/personamem.py`, `eval/benchmarks/personamem_bench.py`
- `repo_pages/benchmarks/personamem/index.md` (current page with the unresolved citation).

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.