agentscope-ai / agentscope-ai/ReMe

benchmark: add BEAM evaluation

Aperta
#355 0 commenti 0 reazioni 1 assegnatario Rivendicata da @xyf2020 Vedi su GitHub
enhancement
Lingua principale
Python
Stelle
3.4k
Fork
298
Merge medio
19h 52m
PR unite (30g)
55

Descrizione

**Background**

[BEAM](https://github.com/mohammadtavakoli78/BEAM) evaluates long-term conversational memory at 128K, 500K, 1M, and 10M-token scales across diverse domains and memory abilities. ReMe needs a scalable BEAM harness to characterize memory quality and resource behavior beyond the context sizes covered by smaller conversational benchmarks.

**Changes**
- Add a BEAM benchmark workflow under `benchmark/beam`
- Import the official conversations, questions, annotations, and evaluation assets without committing downloaded or generated data
- Ingest conversations incrementally and preserve turn order, timestamps, domain, scale, and evidence identifiers
- Support representative smoke-test subsets and full 128K, 500K, 1M, and 10M-token tiers
- Reproduce the official evaluation protocol and report results by memory ability, domain, and context scale
- Measure retrieval evidence quality separately from end-to-end answer quality
- Track ingestion time, index size, query latency, model and embedding usage, token usage, and cost
- Make long runs resumable and add configuration documentation, result schemas, and focused fixture-based tests

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.