kinderp / kinderp/durex

Benchmark context, memory, and token-optimization integrations

Open
#36 0 comments 0 reactions 0 assignees View on GitHub
area:plugins kind:audit
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Parent and milestone

- Parent issue: #24
- Milestone: not scheduled; implementation depends on #14 and starts after #15.

## Problem

TokenSave, RTK, Serena, Graphify, Context7, Repomix, Caveman, LLMLingua, and memory tools solve different problems and publish non-comparable savings claims. Enabling overlapping tools may increase cost or remove critical evidence.

## Outcome

Create a reproducible Durex benchmark that prioritizes task correctness and compares context quality, token use, calls, latency, resources, omissions, branch behavior, privacy, and fallback.

## Scope

- Define representative small, large, typed, dynamic, documentation-heavy, failure-debugging, and branch-change tasks.
- Compare no-plugin baseline with one integration per role.
- Measure correctness, input/output tokens, tool calls, wall time, CPU/RAM/disk, index time, stale data, and raw fallback.
- Review licenses, maintenance, protocol fit, data locality, and supply-chain behavior.
- Recommend one default candidate per role or no default.

## Non-goals

- Combining all candidates in one default stack.
- Accepting project-reported savings without reproduction.
- Optimizing token count at the expense of task success.

## Acceptance criteria

- [ ] Results are reproducible from pinned versions and fixtures.
- [ ] Any lossy omission that changes diagnosis or task outcome fails the candidate.
- [ ] The report distinguishes repository intelligence, docs retrieval, output filtering, prompt compression, and memory.
- [ ] Targeted, failure-path, and conformance tests pass.
- [ ] Architecture, plugin-developer, configuration, and operator documentation is updated.

## Validation

- Run each candidate repeatedly against the same fixtures and publish raw machine-readable results plus a concise decision report.

## Dependencies

#33 defines the measurement hooks; research may start before #15.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with parent issue #24, dependencies #14 and #15, and the measurement hooks defined by #33. Define pinned fixtures and repeated runs for the listed task types, comparing each candidate with the no-plugin baseline. Done means reproducible raw machine-readable results, a concise decision report, passing targeted/failure-path/conformance tests, and updated architecture, plugin-developer, configuration, and operator documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, documentation, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.