agentscope-ai / agentscope-ai/ReMe
benchmark: add BEAM evaluation
- Ngôn ngữ chính
- Python
- Star
- 3.4k
- Fork
- 298
- Merge trung bình
- 19 giờ 52 phút
- Pull request đã merge (30 ngày)
- 55
Mô tả
**Background**
[BEAM](https://github.com/mohammadtavakoli78/BEAM) evaluates long-term conversational memory at 128K, 500K, 1M, and 10M-token scales across diverse domains and memory abilities. ReMe needs a scalable BEAM harness to characterize memory quality and resource behavior beyond the context sizes covered by smaller conversational benchmarks.
**Changes**
- Add a BEAM benchmark workflow under `benchmark/beam`
- Import the official conversations, questions, annotations, and evaluation assets without committing downloaded or generated data
- Ingest conversations incrementally and preserve turn order, timestamps, domain, scale, and evidence identifiers
- Support representative smoke-test subsets and full 128K, 500K, 1M, and 10M-token tiers
- Reproduce the official evaluation protocol and report results by memory ability, domain, and context scale
- Measure retrieval evidence quality separately from end-to-end answer quality
- Track ingestion time, index size, query latency, model and embedding usage, token usage, and cost
- Make long runs resumable and add configuration documentation, result schemas, and focused fixture-based tests
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.