agentscope-ai / agentscope-ai/ReMe
benchmark: complete LongMemEval evaluation
- Linguagem predominante
- Python
- Estrelas
- 3.5k
- Forks
- 298
- Merge médio
- 19h 52min
- PRs com merge (30d)
- 55
Descrição
**Background**
[LongMemEval](https://github.com/xiaowu0162/LongMemEval) evaluates information extraction, multi-session reasoning, knowledge updates, temporal reasoning, and abstention over long, timestamped chat histories. ReMe already contains initial LongMemEval steps and runner scripts, but the workflow should be completed and standardized as a reproducible benchmark.
**Changes**
- Consolidate the existing LongMemEval steps and scripts into a documented end-to-end workflow under `benchmark/longmemeval`
- Support the official cleaned LongMemEval-S, LongMemEval-M, and oracle datasets without committing downloaded data or generated workspaces
- Preserve chronological session ingestion and source identifiers for evidence-level evaluation
- Provide direct-context, retrieval, and agentic-answer baselines using consistent prompts and model settings
- Evaluate the five official memory abilities and report both aggregate and per-category results
- Separate retrieval recall from answer correctness and retain inspectable evidence and judge outputs
- Record dataset revision, configuration, model, embedding, latency, token usage, and cost metadata
- Add resumable runs, deterministic sample subsets, cleanup guidance, and focused tests with fixture data
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Direção de pesquisa
Start in `benchmark/longmemeval` by reading and running the existing LongMemEval steps/runner scripts to map current ingestion, baselines, and scoring flow. Extend it for LongMemEval-S/M/oracle data, preserve session order, split retrieval recall from answer correctness, and emit required metadata; add fixture-based coverage for deterministic subsets, resumable runs, and cleanup. Consider done when this folder has a reproducible documented workflow that produces aggregate and per-category outputs plus inspectable evidence and judge artifacts.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- python
- Domínio
- testing-qa, tooling
- Tipo de issue
- Funcionalidade
- Dificuldade
- 4/5
- Tempo estimado
- 3-5 dias
- Status de atividade
- Pouca atividade
- Clareza
- Razoavelmente clara
- Facilidade para iniciantes
- 33/100