oceanbase / oceanbase/powercontext

feat: evaluate Experience and Skill applicability with Jev

Open
#1,647 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ecosystem enhancement evaluation
Dominant language
Python
Stars
1.1k
Forks
214
Avg merge
1d 11h
Merged PRs (30d)
199

Description

Tracking issue: #1649

Feature description

Evaluate an opt-in Jev selector for the applicability of approved Experience and available Skill revisions to a specific agent task. Support an explicit outcome that none of the candidates applies.

Priority: P2 — selective reuse after the initial Jev integration is evaluated.

Problem and proposed solution

An agent changing an HTTP contract should receive the project's contract-generation experience. An agent only asking what an endpoint does should not automatically load and execute that procedure. Similar names and broad topic matches are insufficient to distinguish the two requests.

Use PowerContext to enumerate or retrieve authorized, eligible revisions. Ask Jev whether each candidate's applicability conditions match the task. For Skills, optionally shortlist by metadata and then read a small number of exact revisions before recommending one. Keep both absolute applicability and relative ranking explicit so the best candidate in a bad shortlist is not treated as useful by default.

Return exact Experience/Skill references and bounded advisory selection. The host remains responsible for loading, permissions, and execution. Begin in the adapter/example layer: the current PreparedContext family selection must not be silently expanded to include Skills.

Acceptance criteria

  • Candidates are restricted to authorized, eligible revisions. Pending, rejected, retired, or incompatible candidates do not become eligible because of a high model score.
  • Selection can abstain. Test near-neighbor skills, explanation-only requests, multiple applicable experiences, incomplete conditions, version/environment mismatch, and no-fit tasks.
  • Preserve complementary experiences where a task needs more than one lesson; do not impose Skill-specific top-one selection on every Artifact family.
  • Compare existing retrieval/host selection with Jev-assisted selection using the same task and candidate set; report applicable-candidate recall, wrong loads, unnecessary loads, task success, and added latency/cost.
  • Pin exact revisions and model/question versions. Test relevant English and Chinese requests and calibrate abstention separately from relative ranking.
  • Recommendation neither installs nor executes a Skill and does not bypass review, publication, package validation, or existing host approval rules.
  • Demonstrate one real host flow with traceable selection and, if the host actually invokes a Skill, existing usage evidence for the invoked revision.

Alternatives considered

Deterministic applicability metadata, embeddings, the existing reranker, and the host model are useful baselines. Jev should remain optional if these approaches are sufficient.

Additional context

This is read-time applicability selection. #1508 covers Experience consolidation and evaluation of proposed Skill revisions; #1635 covers Artifact evolution and proposal attempts. Neither lifecycle is replaced here. Skill recommendations must not require a new public PreparedContext family unless a separate contract design is accepted.

Prepared with AI assistance. Cookbook results are reference material, not reproduced PowerContext results.

Related Jev work: reuse bounded provider and evaluation utilities from #1643, then calibrate applicability and abstention independently.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the adapter/example layer and inspect the current PreparedContext family selection; it must not silently include Skills. Read the current API contract in openapi/powercontext.yaml and review the bounded provider and evaluation utilities from issue #1643. Done means an opt-in, traceable recommendation flow with abstention, exact revisions, evaluation evidence, and no bypass of host approval or execution rules.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.