Benchmark context, memory, and token-optimization integrations
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Parent and milestone
- Parent issue: #24
- Milestone: not scheduled; implementation depends on #14 and starts after #15.
## Problem
TokenSave, RTK, Serena, Graphify, Context7, Repomix, Caveman, LLMLingua, and memory tools solve different problems and publish non-comparable savings claims. Enabling overlapping tools may increase cost or remove critical evidence.
## Outcome
Create a reproducible Durex benchmark that prioritizes task correctness and compares context quality, token use, calls, latency, resources, omissions, branch behavior, privacy, and fallback.
## Scope
- Define representative small, large, typed, dynamic, documentation-heavy, failure-debugging, and branch-change tasks.
- Compare no-plugin baseline with one integration per role.
- Measure correctness, input/output tokens, tool calls, wall time, CPU/RAM/disk, index time, stale data, and raw fallback.
- Review licenses, maintenance, protocol fit, data locality, and supply-chain behavior.
- Recommend one default candidate per role or no default.
## Non-goals
- Combining all candidates in one default stack.
- Accepting project-reported savings without reproduction.
- Optimizing token count at the expense of task success.
## Acceptance criteria
- [ ] Results are reproducible from pinned versions and fixtures.
- [ ] Any lossy omission that changes diagnosis or task outcome fails the candidate.
- [ ] The report distinguishes repository intelligence, docs retrieval, output filtering, prompt compression, and memory.
- [ ] Targeted, failure-path, and conformance tests pass.
- [ ] Architecture, plugin-developer, configuration, and operator documentation is updated.
## Validation
- Run each candidate repeatedly against the same fixtures and publish raw machine-readable results plus a concise decision report.
## Dependencies
#33 defines the measurement hooks; research may start before #15.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with parent issue #24, dependencies #14 and #15, and the measurement hooks defined by #33. Define pinned fixtures and repeated runs for the listed task types, comparing each candidate with the no-plugin baseline. Done means reproducible raw machine-readable results, a concise decision report, passing targeted/failure-path/conformance tests, and updated architecture, plugin-developer, configuration, and operator documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, documentation, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100