ContextLab / ContextLab/llmXive

spec 026: requirement-driven model selection consolidation

Open
#1,285 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## What

Consolidate every model-selection site in the platform into ONE requirement-driven mechanism with a strict boundary between **choosing** a model chain and **executing** it:

```
snap = runtime_snapshot(...) # acquisition — networked, fallible
decision = select(requirement, snap) # selection — pure, deterministic
response = execute(decision.chain, messages, options=..., backend_factory=...)
```

Call sites express **intent** (a named requirement profile), never a model string. Spec:
[`specs/026-model-selection-consolidation/spec.md`](specs/026-model-selection-consolidation/spec.md) ·
plan: [`plan.md`](specs/026-model-selection-consolidation/plan.md) · branch `026-model-selection-consolidation`.

## Relationship to #1242 (traceability)

This issue exists because spec 026 does **NOT** implement #1242's acceptance criteria. #1242
("self-improvement through lessons learned") asks for two cron-driven self-improvement lanes.
Spec 026 is the **model-selection prerequisite** for spec 025's lane, split off as its own spec + PR
by maintainer decision 2026-07-16 (recorded at `spec.md:6`) so it can land first. Work on this branch
should be judged against the criteria below, not against #1242's.

## Acceptance (from spec.md)

- **FR-001/FR-008**: `select()` is a pure function of `(requirement, snapshot, policy_version)`
returning a byte-identical ordered chain; `execute()` never re-ranks.
- **FR-006**: eligibility = versioned capability policy ∩ live snapshot; the catalog proves
presence/price, never competence; unknown cost is never free.
- **FR-010/FR-011/FR-016**: existing router safeguards preserved; paid permission checked twice;
paid calls reserve against the strict-CAS ledger on `refs/llmxive/credit-ledger`.
- **FR-012/FR-013**: all call sites express one of the 6 named profiles; registry agents, the router,
the execution tier-ladder, the catalog filter and every direct pin resolve through the mechanism.
- **FR-014**: a static enforcement test rejects production model-id literals outside the canonical
layer; `DEFAULT_MODEL` + `MODEL_FALLBACKS` are deleted.
- **SC-007**: behavior preserved against a fixed migration snapshot (`qwen.qwen3.5-122b` stays the
default; gemma-3→gemma-4 is the one maintainer-approved correction).
- **SC-008**: real snapshot assembly + real calls across the free tiers, the vision tier, and the
local fallback.

## Status

Slices 1–6 complete (contracts, snapshot+LKG, router primitive, ledger, pure `select`, `execute`,
independence, routing chokepoint, execution-tier migration). Slice 7 (compatibility consumers +
`AgentRegistryEntry` schema migration + US5 AC1/AC4) in design. Slice 8 (static enforcement +
deletions) last.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with specs/026-model-selection-consolidation/spec.md, then read plan.md and the Slice 7 and Slice 8 status. Trace the named profiles, compatibility consumers, AgentRegistryEntry migration, static enforcement, and deletions against the listed acceptance criteria. Done means all model-selection sites use the mechanism, safeguards remain intact, and the migration behavior and real-call criteria pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.