jlevy / jlevy/markform

onToolStart/onToolEnd callbacks never fire for built-in web_search tools

Open
#113 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
70
Forks
3
PR merge metrics
No merged PRs in 30d

Description

## Bug: onToolStart/onToolEnd callbacks never fire for built-in web_search

### Observed

When using `fillForm()` with `enableWebSearch: true` and `FillCallbacks` with
`onToolStart`/`onToolEnd`, only custom `additionalTools` produce per-call callbacks.
Built-in provider web search tools (e.g. `web_search`) appear only in the aggregated
`turnStats.toolCalls` summary (e.g. `web_search(17)`) — no individual
`onToolStart`/`onToolEnd` callbacks fire.

### Expected

`onToolStart` and `onToolEnd` should fire for every tool call, including built-in
provider web search tools, so consumers get per-call visibility (name, input, output,
duration).

### Root Cause

In `liveAgent.ts`, `wrapToolsWithCallbacks()` only wraps tools that have an `execute`
function:

```typescript
const execute = (tool as any).execute;
if (typeof execute === 'function') {
wrapped[name] = wrapTool(name, tool, execute, callbacks);
} else {
// Pass through declarative tools unchanged
wrapped[name] = tool;
}
```

Provider web search tools (loaded via `loadWebSearchTools()`) are **declarative AI SDK
tools** — they have a schema but no `execute` function. The AI SDK handles their
execution internally during `generateText()`, bypassing the callback wrapper entirely.

The aggregated `turnStats.toolCalls` count works because it comes from the AI SDK
response metadata, not from the callback wrapper.

### Suggested Fix

**Option A** (recommended): Hook into the AI SDK's `onStepFinish` or tool result
metadata to emit synthetic `onToolStart`/`onToolEnd` callbacks for declarative tools
after each `generateText()` call.

**Option B**: Convert provider web search tools from declarative to executable wrappers
that call the provider API directly, so `wrapToolsWithCallbacks` can intercept them.

Option A is less invasive and preserves the AI SDK's native handling.

### Reproduction

```typescript
import { fillForm } from 'markform';

const result = await fillForm({
form: myFormMarkdown,
model: 'openai/gpt-5-mini',
enableWebSearch: true,
additionalTools: { my_tool: myTool },
callbacks: {
onToolStart: ({ name, input }) => console.log(`tool start: ${name}`),
onToolEnd: ({ name }) => console.log(`tool end: ${name}`),
},
});
// Logs: "tool start: my_tool", "tool end: my_tool"
// Does NOT log: "tool start: web_search" (despite web_search appearing in turnStats)
```

### Affected Version

markform 0.1.17

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.