oceanbase / oceanbase/powercontext

feat(memory): add an optional Jev reranker with comparative evaluation

Open
#1,643 1 comment 0 reactions 1 assignee View on GitHub

@Oxidaner is already working on this.

Since Sep 19, 2026.

ecosystem enhancement evaluation
Dominant language
Python
Stars
1.1k
Forks
212
Avg merge
1d 11h
Merged PRs (30d)
199

Description

Tracking issue: #1649

Feature description

Add an opt-in Jev implementation of MemoryReranker and evaluate it against the existing retrieval and LLM reranking paths. The intended benefit is to retain useful project evidence at lower latency and cost, without reducing downstream task success.

Priority: P0 — first integration experiment.

Problem and proposed solution

A coding agent can retrieve many entries about a database issue while missing the entry that explains the relevant compatibility constraint. Topic similarity alone does not establish usefulness for the current task.

PowerContext already exposes MemoryReranker and accepts an injected reranker. Implement a bounded adapter that sends the query and eligible candidate text to Jev, asks explicit relevance questions, and converts the resulting scores into validated original candidate ranks. Jev returns typed decisions; replacing the generation model name in the existing listwise generator is not sufficient.

Keep candidate identity, exact citations, authorization, lifecycle eligibility, and final UTF-8 budget enforcement in PowerContext. Start with Memory only. Pin the Jev model version and question-policy version; report usage, latency, and fallback without logging private candidate text by default. Configure a total deadline that includes any SDK retries. On timeout, rate limiting, or unusable results, preserve the original coarse order and report the fallback.

Use a bounded batch or bounded concurrent pair requests based on measured accuracy and latency. Do not assume that packing every candidate into shared state is always better.

Acceptance criteria

  • The adapter implements the existing MemoryReranker contract and is disabled unless explicitly configured. Normal installations do not require Jev credentials.
  • Results contain only unique original candidate ranks within the requested limit; exact evidence identity is preserved through final preparation.
  • Failure and deadline cases return the existing candidate order through a documented fallback. PreparedContext remains read-only and within its existing byte budget.
  • Compare no reranking, the current LLM reranker, and Jev using the same candidate pools, query set, downstream model, and context budget. Include a suitable dedicated reranker baseline if one is available.
  • Cover English and Chinese queries, lexical distractors, complementary evidence, negation, date/version conditions, and queries with no useful candidate.
  • Report relevant-evidence recall, ranking quality, actual injected evidence/bytes, downstream task success, p50/p95 latency, billed input, retries, and total request cost.
  • Validate integration behavior on SQLite and OceanBase, including fallback and authorization boundaries; publish reproducible artifacts and a recommendation to adopt, revise, or stop. Define the quality-regression tolerance before the held-out run.

Alternatives considered

Retaining the existing LLM reranker, using a dedicated reranking model, and improving deterministic retrieval are valid baselines. A lower provider price alone is not sufficient to select Jev.

Additional context

The model-free recall sufficiency gate in #1556 / #1596 is already on master. This issue adds a candidate-ordering implementation; it does not replace the gate or expand the retrieval pool. Keep expansion disabled with reranking unless the deployment explicitly enables and budgets that combination. Broader Memory quality/lifecycle policy remains in #1590; the evaluation framework remains in #1422.

Prepared with AI assistance. This is proposed work; no Jev performance result has been reproduced in PowerContext.

Related Jev evaluations: #1644 assesses evidence coverage; #1646 demonstrates project-aware decisions independently of the reranking experiment.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.