agentscope-ai / agentscope-ai/ReMe
benchmark: add BEAM evaluation
- Lingua principale
- Python
- Stelle
- 3.4k
- Fork
- 298
- Merge medio
- 19h 52m
- PR unite (30g)
- 55
Descrizione
**Background**
[BEAM](https://github.com/mohammadtavakoli78/BEAM) evaluates long-term conversational memory at 128K, 500K, 1M, and 10M-token scales across diverse domains and memory abilities. ReMe needs a scalable BEAM harness to characterize memory quality and resource behavior beyond the context sizes covered by smaller conversational benchmarks.
**Changes**
- Add a BEAM benchmark workflow under `benchmark/beam`
- Import the official conversations, questions, annotations, and evaluation assets without committing downloaded or generated data
- Ingest conversations incrementally and preserve turn order, timestamps, domain, scale, and evidence identifiers
- Support representative smoke-test subsets and full 128K, 500K, 1M, and 10M-token tiers
- Reproduce the official evaluation protocol and report results by memory ability, domain, and context scale
- Measure retrieval evidence quality separately from end-to-end answer quality
- Track ingestion time, index size, query latency, model and embedding usage, token usage, and cost
- Make long runs resumable and add configuration documentation, result schemas, and focused fixture-based tests
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.