Ideas from DSPy: self-improving agents, composable reasoning modules, and LLM call caching

## Context

Deep analysis of [DSPy](https://github.com/stanfordnlp/dspy) (Stanford NLP's "programming — not prompting — language models" framework, 33k stars, v3.1.3) to identify ideas worth borrowing or adapting for the Agents SDK. DSPy's core thesis — prompts should be *derived artifacts* optimized by algorithms, not hand-written strings — maps surprisingly well onto a durable, stateful runtime like ours.

---

## 1. Built-in LLM Call Caching in Durable Storage

**DSPy pattern:** Every LLM call is cached to disk by default (`diskcache` + `xxhash` for cache keys). This makes eval loops reproducible and avoids redundant API costs.

**How it maps to us:** We already have SQLite via Durable Objects (`this.sql`). A transparent cache that hashes `model + messages + tools + temperature` and stores responses in a `cf_agents_llm_cache` table would:

- Eliminate redundant API calls across retries, fibers, and re-evaluations
- Enable the eval/optimization ideas below (they depend on cheap replays)
- Track hit rates via existing observability channels

**Effort:** Low — the storage layer already exists.

---

## 2. Composable Reasoning Modules

**DSPy pattern:** Pre-built modules (`ChainOfThought`, `ReAct`, `BestOfN`, `Refine`, `MultiChainComparison`) that compose like building blocks. Users mix and match instead of reimplementing common patterns.

**How it maps to us:** The `guides/anthropic-patterns/` directory already demonstrates sequential chains, routing, parallel execution, orchestrator, and evaluator patterns — but as one-off guide code. These could be extracted into reusable abstractions:

- **BestOfN** — Run N completions, pick the best by a metric (fibers via `withFibers` already support parallel execution)
- **Refine** — Self-critique loop: generate → evaluate → regenerate if score is low (`Session.onCompaction()` is already this shape)
- **ChainOfThought** — Automatically inject reasoning fields (the `ReasoningUIPart` in `AIChatAgent` already supports this concept)
- **Consensus** — Run with multiple models, take majority vote

These would compose with `onChatMessage` in `AIChatAgent`.

**Effort:** Medium.

---

## 3. Evaluation / Metrics as Runtime Primitives

**DSPy pattern:** `Evaluate` is a first-class citizen. You define a metric, run your program over a dataset, and get a score. Optimizers then search for configurations that maximize that score.

**How it maps to us:** We have `evals/` with evalite, but only as a CI artifact. The idea is making eval a runtime-accessible primitive. Since every Agent is a Durable Object with its own SQLite, it could:

- Log every LLM interaction + outcome
- Accumulate its own performance history
- Expose metrics via the existing observability channels (adding a `metrics` channel alongside `state`, `rpc`, `message`, etc.)

This turns each agent instance into a self-monitoring system.

**Effort:** Medium.

---

## 4. Automatic Context Window Management

**DSPy pattern:** Optimizers manage *what goes into the context window* — which few-shot examples, which instructions, how much. `BootstrapFewShot` selects the best demonstrations from a pool. `KNNFewShot` retrieves nearest-neighbor examples at inference time.

**How it maps to us:** The experimental `Session` class already has the pieces:

- `withContext("soul", { maxTokens: 1100 })` — context blocks with token budgets
- `onCompaction()` — message history compression
- `truncateOlderMessages()` — read-time context truncation

What's missing is **automatic selection**. Instead of the user manually choosing what context to include, the SDK could:

- Maintain a pool of successful interaction examples in SQLite
- Automatically select the most relevant examples per query
- Use observability data to learn which context configurations produce the best outcomes

This turns `Session` from a passive storage layer into an active context optimization engine.

**Effort:** Medium-High.

---

## 5. Durable Prompt Versioning and A/B Testing

**DSPy pattern:** Optimizers produce versioned, serializable "compiled" programs. You can save, load, and compare configurations (`dspy.load("optimized_v3.json")`).

**How it maps to us:** Since every Agent has its own SQLite:

- Track which system prompt / context config / tool set was used per interaction
- Measure outcomes per configuration
- Support A/B testing natively — different agent instances (by name) run different configs
- Roll back to previous configurations if metrics degrade

`Session.freezeSystemPrompt()` / `Session.refreshSystemPrompt()` already handle prompt snapshots. Adding version tracking + metric correlation would turn this into live prompt optimization.

**Effort:** Medium.

---

## 6. Declarative Task Signatures

**DSPy pattern:** You declare *what* a module does (`"question -> answer"`) as a typed signature, not *how* to prompt for it. The framework handles prompt construction, output parsing, and retry logic.

**How it maps to us:** Users of `AIChatAgent` manually construct system prompts, tool descriptions, and wire everything together. A declarative layer where you define capabilities as typed I/O contracts could auto-generate system prompts, tool descriptions, and response parsing:

```typescript
class MyAgent extends AIChatAgent {
@task("question: string -> answer: string")
async answerQuestion(question: string) { /* ... */ }
}
```

**Effort:** High — requires a new abstraction layer.

---

## 7. Self-Improving Agents (The Big One)

**The synthesis of DSPy's optimization philosophy with our durable runtime.**

DSPy runs optimization offline and deploys the result. Because our agents are **durable** — they persist state across requests, have SQLite, survive evictions — they can optimize themselves at runtime:

1. Log every LLM interaction + outcome in durable SQLite (see #3 above)
2. Periodically (via `this.schedule()`) run a self-improvement step:
- Analyze which system prompt variants produced the best outcomes
- Select the best few-shot examples from recent successful interactions
- Adjust tool descriptions based on tool-call success rates
3. Apply the improved configuration for subsequent requests

No offline pipeline required. Each agent instance literally gets better over time.

This is something no other framework can do because nobody else has per-agent persistent state at this level. DSPy proves the optimization algorithms work; our runtime provides the durable substrate to run them continuously.

**Effort:** High — builds on all the above.

---

## Suggested Priority

| Idea | Effort | Impact | Builds on |
|------|--------|--------|-----------|
| LLM call caching in SQLite | Low | High | `this.sql`, Session |
| Composable reasoning modules (BestOfN, Refine) | Medium | High | Fibers, AIChatAgent |
| Eval/metrics as runtime primitives | Medium | High | Observability channels |
| Automatic context selection | Medium-High | High | Session, context blocks |
| Durable prompt versioning + A/B | Medium | Medium | Session.freezeSystemPrompt |
| Declarative task signatures | High | High | New abstraction |
| Self-improving agents | High | Very High | All of the above |

## References

- [DSPy repo](https://github.com/stanfordnlp/dspy)
- [DSPy paper (ICLR 2024)](https://arxiv.org/abs/2310.03714)
- [GEPA: Reflective Prompt Evolution (Jul 2025)](https://arxiv.org/abs/2507.19457)
- [DSPy docs](https://dspy.ai)