oceanbase / oceanbase/powercontext
feat: evaluate Experience and Skill applicability with Jev
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 214
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 199
Description
Tracking issue: #1649
Feature description
Evaluate an opt-in Jev selector for the applicability of approved Experience and available Skill revisions to a specific agent task. Support an explicit outcome that none of the candidates applies.
Priority: P2 — selective reuse after the initial Jev integration is evaluated.
Problem and proposed solution
An agent changing an HTTP contract should receive the project's contract-generation experience. An agent only asking what an endpoint does should not automatically load and execute that procedure. Similar names and broad topic matches are insufficient to distinguish the two requests.
Use PowerContext to enumerate or retrieve authorized, eligible revisions. Ask Jev whether each candidate's applicability conditions match the task. For Skills, optionally shortlist by metadata and then read a small number of exact revisions before recommending one. Keep both absolute applicability and relative ranking explicit so the best candidate in a bad shortlist is not treated as useful by default.
Return exact Experience/Skill references and bounded advisory selection. The host remains responsible for loading, permissions, and execution. Begin in the adapter/example layer: the current PreparedContext family selection must not be silently expanded to include Skills.
Acceptance criteria
- Candidates are restricted to authorized, eligible revisions. Pending, rejected, retired, or incompatible candidates do not become eligible because of a high model score.
- Selection can abstain. Test near-neighbor skills, explanation-only requests, multiple applicable experiences, incomplete conditions, version/environment mismatch, and no-fit tasks.
- Preserve complementary experiences where a task needs more than one lesson; do not impose Skill-specific top-one selection on every Artifact family.
- Compare existing retrieval/host selection with Jev-assisted selection using the same task and candidate set; report applicable-candidate recall, wrong loads, unnecessary loads, task success, and added latency/cost.
- Pin exact revisions and model/question versions. Test relevant English and Chinese requests and calibrate abstention separately from relative ranking.
- Recommendation neither installs nor executes a Skill and does not bypass review, publication, package validation, or existing host approval rules.
- Demonstrate one real host flow with traceable selection and, if the host actually invokes a Skill, existing usage evidence for the invoked revision.
Alternatives considered
Deterministic applicability metadata, embeddings, the existing reranker, and the host model are useful baselines. Jev should remain optional if these approaches are sufficient.
Additional context
This is read-time applicability selection. #1508 covers Experience consolidation and evaluation of proposed Skill revisions; #1635 covers Artifact evolution and proposal attempts. Neither lifecycle is replaced here. Skill recommendations must not require a new public PreparedContext family unless a separate contract design is accepted.
- TypeSafe Skill suggestion cookbook
- Choice and Noul do not share interchangeable thresholds
- Current API contract
Prepared with AI assistance. Cookbook results are reference material, not reproduced PowerContext results.
Related Jev work: reuse bounded provider and evaluation utilities from #1643, then calibrate applicability and abstention independently.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the adapter/example layer and inspect the current PreparedContext family selection; it must not silently include Skills. Read the current API contract in openapi/powercontext.yaml and review the bounded provider and evaluation utilities from issue #1643. Done means an opt-in, traceable recommendation flow with abstention, exact revisions, evaluation evidence, and no bypass of host approval or execution rules.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100