agentscope-ai / agentscope-ai/ReMe

benchmark: complete LongMemEval evaluation

Aberta
#354 0 comentários 0 reações 1 responsável Reivindicada por @xyf2020 Ver no GitHub
enhancement
Linguagem predominante
Python
Estrelas
3.5k
Forks
298
Merge médio
19h 52min
PRs com merge (30d)
55

Descrição

**Background**

[LongMemEval](https://github.com/xiaowu0162/LongMemEval) evaluates information extraction, multi-session reasoning, knowledge updates, temporal reasoning, and abstention over long, timestamped chat histories. ReMe already contains initial LongMemEval steps and runner scripts, but the workflow should be completed and standardized as a reproducible benchmark.

**Changes**
- Consolidate the existing LongMemEval steps and scripts into a documented end-to-end workflow under `benchmark/longmemeval`
- Support the official cleaned LongMemEval-S, LongMemEval-M, and oracle datasets without committing downloaded data or generated workspaces
- Preserve chronological session ingestion and source identifiers for evidence-level evaluation
- Provide direct-context, retrieval, and agentic-answer baselines using consistent prompts and model settings
- Evaluate the five official memory abilities and report both aggregate and per-category results
- Separate retrieval recall from answer correctness and retain inspectable evidence and judge outputs
- Record dataset revision, configuration, model, embedding, latency, token usage, and cost metadata
- Add resumable runs, deterministic sample subsets, cleanup guidance, and focused tests with fixture data

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Direção de pesquisa

Start in `benchmark/longmemeval` by reading and running the existing LongMemEval steps/runner scripts to map current ingestion, baselines, and scoring flow. Extend it for LongMemEval-S/M/oracle data, preserve session order, split retrieval recall from answer correctness, and emit required metadata; add fixture-based coverage for deterministic subsets, resumable runs, and cleanup. Consider done when this folder has a reproducible documented workflow that produces aggregate and per-category outputs plus inspectable evidence and judge artifacts.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
python
Domínio
testing-qa, tooling
Tipo de issue
Funcionalidade
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Pouca atividade
Clareza
Razoavelmente clara
Facilidade para iniciantes
33/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.