cloudflare / cloudflare/mcp

Add evals for search and execute tools

Open
#68 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
833
Forks
116
Avg merge
7h 19m
Merged PRs (30d)
3

Description

## Summary

Add an evaluation suite to validate that AI models can correctly use the `search` and `execute` tools exposed by the MCP server. Multiple models (Workers AI, OpenAI, Anthropic, Google) should be tested to ensure broad compatibility.

The key insight: we can subclass `GlobalOutbound` to spy on the actual `fetch` calls made by LLM-generated code. This gives us deterministic assertions on what endpoints were hit, and per-test control over mock responses — all inside workerd, no MSW needed.

## Architecture

### Spy on GlobalOutbound.fetch

Every `fetch` from LLM-generated code flows through `GlobalOutbound.fetch()`. We subclass it with a `TestGlobalOutbound` that:

1. **Records all outbound requests** (method, url, body)
2. **Matches against registered handlers** to return realistic per-endpoint responses
3. **Falls back** to a generic success response for unhandled routes

This means:
- **LLMs still do the hard part** — understanding prompts, searching the spec, writing `cloudflare.request()` code
- **Assertions are deterministic** — check the actual outbound request method/path/body
- **Responses are realistic** — per-test handlers return endpoint-appropriate data so LLM code doesn't choke
- **No production code changes** — just bind `TestGlobalOutbound` instead of `GlobalOutbound` in eval config
- **Runs inside workerd** — no MSW needed, the spy lives where the fetch happens

### TestGlobalOutbound

```ts
// evals/test-outbound.ts
import { GlobalOutbound } from '../src/index'

type MockHandler = (req: { method: string; url: string; body?: string }) => Response

export class TestGlobalOutbound extends GlobalOutbound {
static requests: { method: string; url: string; body?: string }[] = []
static handlers: Map = new Map()

static reset() {
this.requests = []
this.handlers.clear()
}

static on(pattern: string, handler: MockHandler) {
this.handlers.set(pattern, handler)
}

async fetch(request: Request): Promise {
const url = new URL(request.url)
const entry = {
method: request.method,
url: url.pathname + url.search,
body: request.body ? await request.clone().text() : undefined,
}
TestGlobalOutbound.requests.push(entry)

// Match against registered handlers
for (const [pattern, handler] of TestGlobalOutbound.handlers) {
if (entry.url.includes(pattern)) {
return handler(entry)
}
}

// Default fallback
return Response.json({
success: true,
result: [],
errors: [],
messages: [],
})
}
}
```

## Proposed Structure

```
evals/
├── test-outbound.ts # TestGlobalOutbound spy
├── utils.ts # Client setup, model config
├── search.eval.ts # Search tool evals
├── execute.eval.ts # Execute tool evals
└── end-to-end.eval.ts # Search → execute flows
```

### Vitest config (`vitest.config.evals.ts`)

```ts
import { defineWorkersConfig } from '@cloudflare/vitest-pool-workers/config'

export default defineWorkersConfig({
test: {
include: ['evals/**/*.eval.?(c|m)[jt]s?(x)'],
poolOptions: {
workers: {
isolatedStorage: true,
wrangler: { configPath: './wrangler.jsonc' },
miniflare: {
bindings: {
ENVIRONMENT: 'test',
},
},
},
},
},
})
```

### Package.json scripts

```json
{
"eval:dev": "start-server-and-test --expect 404 eval:server http://localhost:8978 'vitest --testTimeout=60000 --config vitest.config.evals.ts'",
"eval:server": "wrangler dev --var ENVIRONMENT:test --var DEV_DISABLE_OAUTH:true --port 8978",
"eval:ci": "start-server-and-test --expect 404 eval:server http://localhost:8978 'vitest run --testTimeout=60000 --config vitest.config.evals.ts'"
}
```

## Code Examples

### `evals/utils.ts`

```ts
import { MCPClientManager } from 'agents/mcp/client'

export async function initializeClient(): Promise {
const clientManager = new MCPClientManager('eval-client', '0.0.0')
await clientManager.connect('http://localhost:8978/sse')
return clientManager
}
```

### `evals/search.eval.ts`

```ts
import { describe, it, expect } from 'vitest'
import { runTask } from '@repo/eval-tools/src/runTask'
import { eachModel } from '@repo/eval-tools/src/test-models'
import { initializeClient } from './utils'

eachModel('$modelName', ({ model }) => {
describe('search tool', () => {
it('finds Workers endpoints', async () => {
const client = await initializeClient()
const { toolCalls } = await runTask(
client,
model,
'Find all Cloudflare Workers script endpoints'
)

const searchCall = toolCalls.find((call) => call.toolName === 'search')
expect(searchCall, 'search tool was not called').toBeDefined()
})

it('finds KV namespace endpoints', async () => {
const client = await initializeClient()
const { toolCalls } = await runTask(
client,
model,
'Find the API endpoint to create a new KV namespace'
)

const searchCall = toolCalls.find((call) => call.toolName === 'search')
expect(searchCall, 'search tool was not called').toBeDefined()
})
})
})
```

### `evals/execute.eval.ts`

```ts
import { describe, it, expect } from 'vitest'
import { runTask } from '@repo/eval-tools/src/runTask'
import { eachModel } from '@repo/eval-tools/src/test-models'
import { TestGlobalOutbound } from './test-outbound'
import { initializeClient } from './utils'

eachModel('$modelName', ({ model }) => {
describe('execute tool', () => {
it('lists Workers scripts', async () => {
TestGlobalOutbound.reset()
TestGlobalOutbound.on('/workers/scripts', () =>
Response.json({
success: true,
result: [{ id: 'my-worker', etag: 'abc123', modified_on: '2025-01-01' }],
result_info: { page: 1, per_page: 20, count: 1, total_count: 1 },
errors: [],
messages: [],
})
)

const client = await initializeClient()
await runTask(client, model, 'List all my Cloudflare Workers scripts')

expect(TestGlobalOutbound.requests).toEqual(
expect.arrayContaining([
expect.objectContaining({
method: 'GET',
url: expect.stringMatching(/\/accounts\/[^/]+\/workers\/scripts/),
}),
])
)
})

it('lists KV namespaces', async () => {
TestGlobalOutbound.reset()
TestGlobalOutbound.on('/storage/kv/namespaces', () =>
Response.json({
success: true,
result: [
{ id: 'ns-1', title: 'MY_KV', supports_url_encoding: true },
{ id: 'ns-2', title: 'SESSIONS', supports_url_encoding: true },
],
result_info: { page: 1, per_page: 20, count: 2, total_count: 2 },
errors: [],
messages: [],
})
)

const client = await initializeClient()
await runTask(client, model, 'List all my KV namespaces')

expect(TestGlobalOutbound.requests).toEqual(
expect.arrayContaining([
expect.objectContaining({
method: 'GET',
url: expect.stringMatching(/\/accounts\/[^/]+\/storage\/kv\/namespaces/),
}),
])
)
})

it('creates a D1 database with the right name', async () => {
TestGlobalOutbound.reset()
TestGlobalOutbound.on('/d1/database', () =>
Response.json({
success: true,
result: { uuid: 'db-123', name: 'my-db', created_at: '2025-01-01' },
errors: [],
messages: [],
})
)

const client = await initializeClient()
await runTask(client, model, 'Create a new D1 database called "my-db"')

expect(TestGlobalOutbound.requests).toEqual(
expect.arrayContaining([
expect.objectContaining({
method: 'POST',
url: expect.stringMatching(/\/accounts\/[^/]+\/d1\/database/),
body: expect.stringContaining('my-db'),
}),
])
)
})
})
})
```

### `evals/end-to-end.eval.ts`

```ts
import { describe, it, expect } from 'vitest'
import { runTask } from '@repo/eval-tools/src/runTask'
import { eachModel } from '@repo/eval-tools/src/test-models'
import { TestGlobalOutbound } from './test-outbound'
import { initializeClient } from './utils'

eachModel('$modelName', ({ model }) => {
describe('search then execute', () => {
it('finds and calls the DNS zones endpoint', async () => {
TestGlobalOutbound.reset()
TestGlobalOutbound.on('/zones', () =>
Response.json({
success: true,
result: [
{ id: 'zone-1', name: 'example.com', status: 'active' },
{ id: 'zone-2', name: 'test.dev', status: 'active' },
],
result_info: { page: 1, per_page: 20, count: 2, total_count: 2 },
errors: [],
messages: [],
})
)

const client = await initializeClient()
const { toolCalls } = await runTask(
client,
model,
'Find the endpoint to list DNS zones, then call it'
)

// Model used search first
const searchCall = toolCalls.find((call) => call.toolName === 'search')
expect(searchCall, 'search tool was not called').toBeDefined()

// Then actually hit the right endpoint
expect(TestGlobalOutbound.requests).toEqual(
expect.arrayContaining([
expect.objectContaining({
method: 'GET',
url: expect.stringMatching(/\/zones/),
}),
])
)
})
})
})
```

## Suggested eval coverage

- **Search tool**: find Workers endpoints, find KV endpoints, find DNS endpoints, search by HTTP method
- **Execute tool**: list Workers scripts, list KV namespaces, create D1 database, POST with body, query parameters
- **End-to-end**: search → execute flows, multi-step operations

## Models to test

Use `eachModel` to run across:
- Workers AI models (via `workers-ai-provider`)
- OpenAI (GPT-4o, GPT-4o-mini)
- Anthropic (Claude Sonnet)
- Google (Gemini Flash)

Route through AI Gateway for unified API management.

## Notes

- The `eval-tools` package from `mcp-server-cloudflare` (`runTask`, `eachModel`) can be extracted into a shared package or vendored. We don't need `checkFactuality` / `describeEval` / `vitest-evals` — the outbound spy gives us deterministic assertions with plain vitest.
- `TestGlobalOutbound` subclasses `GlobalOutbound` and binds in via the executor's existing `globalOutbound` parameter — no changes to production code needed.
- Each test registers its own handlers with `TestGlobalOutbound.on()` so responses are realistic and endpoint-appropriate.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.