[Agentic Interface] Add VCR-Style Agent Conversation Testing to `example-mcp-dashbuilder`

Depends on #33.

This concept is inspired by something similar I created for `eddo` here: https://github.com/walterra/eddoapp/pull/73

Integration tests verify individual MCP tools are correct and deterministic — they call `create_chart`, `run_esql`, etc. directly and assert responses. But they don't test the **end-to-end agent workflow**: an LLM reading resources, deciding which tools to call, chaining multi-step sequences (explore → query → create chart → organize → export), and producing a coherent dashboard.

Testing this live is expensive (LLM tokens), slow (API latency), and non-deterministic (LLM output varies). We need a way to:

1. **Record** a real LLM-driven agent conversation once (the LLM's decisions, tool calls, and final output).
2. **Replay** the LLM side from the recording, while MCP tool calls execute for real against a live server + Elasticsearch — verifying the tools still produce correct results for the same conversation flow.

This is complementary to integration tests, not a replacement.

## Goals

1. Record full agent conversations (LLM prompts, LLM responses, tool calls, tool results) to cassette files.
2. On replay, substitute the LLM with cached responses — the MCP server and Elasticsearch remain real.
3. Detect when tool results diverge from the recording (schema changes, query behavior changes).
4. Support three modes: `auto`, `record`, `playback`.
5. Keep integration tests unchanged — VCR is a separate test suite.

## How It Works

```
┌─────────────────────────────────────────────────────┐
│ RECORD mode │
│ │
│ Test Script → Real LLM API → MCP Server (real) │
│ ↓ ↓ │
│ save responses execute tools (real ES)│
│ ↓ │
│ cassette.json │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│ PLAYBACK mode │
│ │
│ Test Script → Cached LLM → MCP Server (real) │
│ (from cassette) ↓ │
│ execute tools (real ES)│
│ ↓ │
│ verify results match │
└─────────────────────────────────────────────────────┘
```

In **record** mode, a real LLM (e.g. Claude) drives the conversation — reading MCP resources, calling tools, receiving results, deciding next steps. Every LLM request/response pair is saved to a cassette file.

In **playback** mode, the LLM is replaced by the cassette — it returns the same responses it gave during recording. But the MCP server is real, tool calls execute against real Elasticsearch, and results flow back. If tool results have changed (e.g. a schema change broke `create_chart`), the test catches it.

## Design

### Directory structure

```
server/src/vcr-tests/
├── vcr/
│ ├── index.ts # Public API
│ ├── cassette-types.ts # Types for cassette format
│ ├── cassette-manager.ts # Record/replay orchestrator
│ ├── cassette-helpers.ts # Hashing, normalization, file I/O
│ └── cached-llm-service.ts # LLM service that records/replays via cassettes
├── cassettes/ # Recorded conversations (committed to git)
│ ├── build-ecommerce-dashboard.json
│ ├── import-and-modify-kibana-dashboard.json
│ └── explore-data-create-metrics.json
├── helpers/
│ └── agent-harness.ts # Orchestrates LLM ↔ MCP conversation loop
└── suites/
├── dashboard-creation.test.ts
├── data-exploration.test.ts
└── kibana-roundtrip.test.ts
```

### 1. Cassette format: `vcr/cassette-types.ts`

```typescript
/** A single LLM interaction in the conversation */
export interface LLMInteraction {
requestHash: string;
request: {
model: string;
systemPrompt: string;
messages: Array<{ role: string; content: string }>;
};
response: string; // Raw LLM response text (includes tool_use blocks)
metadata: {
recordedAt: string;
responseTimeMs: number;
};
}

/** Full recorded conversation */
export interface Cassette {
version: 1;
testName: string;
createdAt: string;
frozenTime: string; // Timestamp for deterministic time-dependent behavior
interactions: LLMInteraction[];
}

export type RecordMode = 'record' | 'playback' | 'auto';
```

### 2. Request normalization: `vcr/cassette-helpers.ts`

LLM requests contain dynamic content (tool result timestamps, IDs, row counts) that changes between runs even when the conversation flow is identical. Normalization ensures stable hashing:

```typescript
/** Normalize dynamic content in messages for stable hashing */
function normalizeMessageContent(content: string): string {
return content
// Normalize ISO timestamps from tool results
.replace(/\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z/g, '[ISO_DATE]')
// Normalize dashboard IDs (generated per session)
.replace(/"dashboardId"\s*:\s*"[^"]+"/g, '"dashboardId":"[DASHBOARD_ID]"')
// Normalize chart IDs
.replace(/"id"\s*:\s*"[a-z0-9-]+-[a-z0-9]{4}"/g, '"id":"[CHART_ID]"')
// Normalize numeric values that may have floating-point variance
.replace(/\d+\.\d{6,}/g, '[FLOAT]');
}

/** Hash an LLM request for cassette matching */
export function hashRequest(
model: string,
systemPrompt: string,
messages: Array<{ role: string; content: string }>,
): string {
const normalized = JSON.stringify({
model,
systemPrompt: normalizeSystemPrompt(systemPrompt),
messages: messages.map(m => ({
role: m.role,
content: normalizeMessageContent(m.content),
})),
});
return createHash('sha256').update(normalized).digest('hex').substring(0, 16);
}
```

### 3. Cassette manager: `vcr/cassette-manager.ts`

Core record/replay logic — same pattern as the types above but handling the conversation-level lifecycle:

- `loadCassette(testName)` — load existing cassette or create empty one
- `handleInteraction(model, systemPrompt, messages, realCall)` — replay from cassette if hash matches, otherwise call real LLM and record
- `ejectCassette()` — save if modified, reset state

On **hash mismatch** (tool results changed enough to alter the conversation), the cassette truncates at the mismatch point. In `auto` mode, it re-records from that point using the real LLM. In `playback` mode, it fails with a descriptive error showing what diverged.

### 4. Cached LLM service: `vcr/cached-llm-service.ts`

Wraps a real LLM client (e.g. Anthropic SDK) with the cassette manager:

```typescript
export function createCachedLLMService(config: {
cassetteManager: CassetteManager;
model?: string;
}) {
const { cassetteManager, model = 'claude-sonnet-4-20250514' } = config;

// Lazy-init real client only when recording
let realClient: Anthropic | null = null;
function getClient(): Anthropic {
if (!realClient) {
if (!process.env.ANTHROPIC_API_KEY) {
throw new Error('ANTHROPIC_API_KEY required for recording. Use VCR_MODE=playback for cached responses.');
}
realClient = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
}
return realClient;
}

async function generateResponse(
systemPrompt: string,
messages: Array<{ role: string; content: string }>,
): Promise {
return cassetteManager.handleInteraction(model, systemPrompt, messages, async () => {
const response = await getClient().messages.create({
model,
max_tokens: 4096,
system: systemPrompt,
messages: messages as Anthropic.MessageParam[],
});
// Extract text content
const text = response.content.find(c => c.type === 'text');
return text?.text ?? '';
});
}

return { generateResponse };
}
```

### 5. Agent harness: `helpers/agent-harness.ts`

Orchestrates the LLM ↔ MCP conversation loop — the agent loop that reads resources, calls tools, and iterates:

```typescript
export class AgentHarness {
constructor(
private llmService: { generateResponse: (system: string, messages: Message[]) => Promise },
private mcpServer: MCPTestServer,
) {}

/** Run a full agent conversation from a user prompt */
async run(userPrompt: string, maxTurns = 20): Promise {
// 1. Read MCP resources (dataviz guidelines, esql reference)
// 2. Build system prompt from server instructions + resources
// 3. Loop: send messages to LLM → parse tool calls → execute via MCP → append results → repeat
// 4. Return final conversation state + dashboard state
}
}
```

The harness parses tool_use blocks from LLM responses, calls `mcpServer.callTool()` for each, and feeds results back as conversation messages — exactly how a real MCP client works.

### 6. Test suites

**`dashboard-creation.test.ts`** — record a conversation where the LLM builds a complete dashboard:

```typescript
describe('VCR: Dashboard Creation', () => {
let vcr: CassetteManager;
let testServer: MCPTestServer;
let agent: AgentHarness;

beforeEach(async () => {
vcr = createCassetteManager({
cassettesDir: resolve(__dirname, '../cassettes'),
mode: (process.env.VCR_MODE as RecordMode) || 'auto',
});
vcr.loadCassette(expect.getState().currentTestName!);

testServer = new MCPTestServer();
await testServer.start();

const llm = createCachedLLMService({ cassetteManager: vcr });
agent = new AgentHarness(llm, testServer);
});

afterEach(async () => {
vcr.ejectCassette();
await testServer.stop();
});

it('should build an ecommerce dashboard from a natural language prompt', async () => {
const result = await agent.run(
'Create a dashboard for the kibana_sample_data_ecommerce index showing revenue trends, top categories, and order count.'
);

// The LLM's decisions are replayed, but tools execute for real
expect(result.toolCalls).toContainEqual(expect.objectContaining({ name: 'create_dashboard' }));
expect(result.toolCalls).toContainEqual(expect.objectContaining({ name: 'create_chart' }));
expect(result.errors).toHaveLength(0);

// Verify dashboard state via MCP
const dashboard = await testServer.callTool('view_dashboard', {
dashboardId: result.dashboardId,
});
expectNoError(dashboard);
});
});
```

**`data-exploration.test.ts`** — LLM explores indices, runs queries, then creates visualizations.

**`kibana-roundtrip.test.ts`** — LLM imports a Kibana dashboard, modifies it, exports back (optional, needs Kibana).

### 7. Vitest config: `server/vitest.vcr.config.ts`

Separate from integration tests — VCR tests have different timeout and dependency requirements:

```typescript
export default defineConfig({
test: {
name: 'mcp-vcr',
include: ['src/vcr-tests/suites/**/*.test.ts'],
testTimeout: 120_000, // LLM calls can be slow when recording
hookTimeout: 60_000,
pool: 'forks',
poolOptions: { forks: { singleFork: true } },
maxConcurrency: 1, // Sequential — cassettes are order-dependent
reporters: ['verbose'],
},
});
```

### 8. Scripts

```json
{
"scripts": {
"test:vcr": "vitest run --config vitest.vcr.config.ts",
"test:vcr:record": "VCR_MODE=record vitest run --config vitest.vcr.config.ts",
"test:vcr:playback": "VCR_MODE=playback vitest run --config vitest.vcr.config.ts"
}
}
```

### 9. Environment variables

| Variable | Default | Description |
|---|---|---|
| `VCR_MODE` | `auto` | `auto` = record if missing, replay if exists; `record` = always re-record; `playback` = replay only, fail if cassette missing |
| `VCR_DEBUG` | _(unset)_ | Log cassette operations (load, replay, record, hash match/mismatch) |
| `ANTHROPIC_API_KEY` | _(required for record)_ | LLM API key — only needed when recording |
| `ELASTICSEARCH_URL` | _(required)_ | Real ES instance — needed for both record and playback (tools execute for real) |

## Relationship to Integration Tests

| | Integration Tests | VCR Tests |
|---|---|---|
| **What's tested** | Individual tool correctness | Full agent conversation flows |
| **LLM involved** | No | Yes (cached on replay) |
| **ES required** | Yes (testcontainer) | Yes (testcontainer) |
| **Deterministic** | Fully | LLM side is cached; tool side is real |
| **Cost** | Free | Free on replay; LLM tokens on record |
| **Speed** | Fast (~30s) | Fast on replay (~30s); slow on record (LLM latency) |
| **Catches** | Tool regressions, schema errors | Workflow regressions, tool chaining issues, response format changes that break LLM expectations |

## Scope

### In scope
- `server/src/vcr-tests/vcr/` — VCR module (5 files)
- `server/src/vcr-tests/cassettes/` — recorded conversations (committed to git)
- `server/src/vcr-tests/helpers/agent-harness.ts` — LLM ↔ MCP conversation loop
- `server/src/vcr-tests/suites/` — VCR test suites
- `server/vitest.vcr.config.ts` — separate vitest config
- `server/package.json` — add `@anthropic-ai/sdk` as devDependency, add `test:vcr*` scripts

### Out of scope
- Multi-model support (record with one LLM, replay is model-agnostic)
- Cassette auto-expiry or staleness detection
- Streaming LLM responses (record full responses only)
- Preview app / UI testing

## Acceptance Criteria

- [ ] `VCR_MODE=record` with real LLM + real ES records a multi-turn conversation cassette
- [ ] `VCR_MODE=playback` replays the LLM side from cassette while MCP tools execute against real ES
- [ ] Tool results that diverge from recording are detected and logged
- [ ] Hash mismatch in `auto` mode triggers re-recording from the divergence point
- [ ] `playback` mode fails with descriptive error when cassette is missing or hash mismatches
- [ ] `ANTHROPIC_API_KEY` is only required in `record` mode
- [ ] Cassettes are human-readable JSON with truncated message bodies
- [ ] `VCR_DEBUG=1` logs replay/record decisions to stderr
- [ ] Integration tests (`test:integration`) are completely unaffected