jlevy / jlevy/markform

feat: extensible provider support via ProviderAdapter registration

Open
#136 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
70
Forks
3
PR merge metrics
No merged PRs in 30d

Description

## Summary

Markform currently hard-codes 5 AI SDK providers (anthropic, openai, google, xai, deepseek) in `modelResolver.ts`. Passing a model string like `deepinfra/glm-5` to `fillForm()` throws `Unknown provider: "deepinfra"`. This makes it impossible to use any other AI SDK provider without workarounds.

The proposal is to add a `ProviderAdapter` interface and a `registerProvider()` function that lets callers bring their own `@ai-sdk/*` providers while preserving the existing 5 built-in providers as defaults.

## Motivation (Real-World Example)

We are building a batch research pipeline that runs markform form fills across multiple LLM providers for comparison. We needed to test models hosted on DeepInfra (GLM-5, Kimi K2.5) alongside the built-in providers.

Since markform does not support DeepInfra, we had to:

1. Create a separate `modelResolver.ts` in our codebase that imports `@ai-sdk/deepinfra`, creates a `LanguageModel` instance, and passes it to `fillForm()`
2. Add a `resolvedModel?: LanguageModel` option to our markform runner wrapper
3. Use `model: resolvedModel ?? modelIdString` to bypass markform string resolution

This workaround has drawbacks:
- When `LanguageModel` is passed directly, markform sets `provider = undefined`, disabling web search and losing provider metadata in fill records
- In parallel execution, each scoped agent needs the LanguageModel -- if batch orchestration crosses process boundaries, the LanguageModel (which contains closures) cannot be serialized
- Every consumer of markform that wants a non-default provider must implement the same workaround

## Proposed Design

### 1. `ProviderAdapter` Interface

```typescript
/**
* Adapter for an AI provider. Clients import their own @ai-sdk/* package,
* configure it, and pass the adapter to markform.
*/
interface ProviderAdapter {
/** Resolve a model name to a LanguageModel instance */
model(modelId: string): LanguageModel;
/** Optional provider-specific tools (web search, etc.) */
tools?: Record;
}
```

### 2. Accept AI SDK Providers Directly

AI SDK provider instances are callable `(modelId: string) => LanguageModel` and may have a `.tools` property. Markform should accept either shape and normalize internally:

```typescript
// Accept AI SDK provider callables OR explicit adapters
type ProviderInput =
| ProviderAdapter
| ((modelId: string) => LanguageModel);

function normalizeProvider(input: ProviderInput): ProviderAdapter {
if (typeof input === 'function') {
const fn = input as ((id: string) => LanguageModel) & {
tools?: Record Tool>;
};
return {
model: (id) => fn(id),
tools: autoExtractTools(fn.tools),
};
}
return input;
}
```

### 3. Registration API

```typescript
// Global registration (persists across fillForm calls)
export function registerProvider(name: string, provider: ProviderInput): void;
export function unregisterProvider(name: string): void;
export function getProviders(): string[];
```

### 4. Per-Call Providers in FillOptions

```typescript
interface FillOptions {
model: string | LanguageModel;
/** Additional providers for string-based model resolution */
providers?: Record;
// ... everything else unchanged
}
```

### 5. Resolution Priority

```
1. options.providers[name] --> per-call override (highest priority)
2. globalRegistry.get(name) --> registerProvider() (session-level)
3. BUILT_IN_PROVIDERS[name] --> current 5 defaults (preserved as-is)
4. --> actionable error with hint
```

The existing 5 built-in providers (anthropic, openai, google, xai, deepseek) continue to work exactly as they do today -- no breaking changes. The new API only extends what is possible.

### 6. Web Search Tool Auto-Extraction

When an AI SDK provider callable is passed directly, markform can duck-type the `.tools` property to auto-detect web search:

```typescript
function autoExtractTools(
providerTools?: Record Tool>,
): Record | undefined {
if (!providerTools) return undefined;
const tools: Record = {};
for (const name of ['webSearch', 'webSearch_20250305', 'googleSearch']) {
if (typeof providerTools[name] === 'function') {
tools['web_search'] = providerTools[name]({});
break;
}
}
return Object.keys(tools).length > 0 ? tools : undefined;
}
```

## Usage Examples

### Zero-Friction AI SDK Provider

```typescript
import { createDeepInfra } from '@ai-sdk/deepinfra';

// One-liner global registration
registerProvider('deepinfra', createDeepInfra({ apiKey }));

// Now string-based model IDs just work
await fillForm({
model: 'deepinfra/glm-5',
enableWebSearch: false,
// ...
});
```

### Per-Call Provider

```typescript
import { createDeepInfra } from '@ai-sdk/deepinfra';

await fillForm({
model: 'deepinfra/glm-5',
providers: {
deepinfra: createDeepInfra({ apiKey }),
},
// ...
});
```

### Custom Adapter with Model Name Mapping

```typescript
const DEEPINFRA_MODEL_MAP: Record = {
'glm-5': 'zai-org/GLM-5',
'kimi-k2.5': 'moonshotai/Kimi-K2.5',
};

registerProvider('deepinfra', {
model: (id) => deepinfra(DEEPINFRA_MODEL_MAP[id] ?? id),
});
```

### AI SDK Provider with Auto-Detected Web Search

```typescript
import { createOpenAI } from '@ai-sdk/openai';

// Web search tools auto-extracted from openai.tools.webSearch
registerProvider('openai', createOpenAI({ apiKey }));

await fillForm({
model: 'openai/gpt-5-mini',
enableWebSearch: true, // auto-detected from provider.tools
});
```

## Why This Design

- **Static dependencies**: Client owns the `@ai-sdk/*` import -- bundler validates, tree-shakes, and type-checks at build time. No dynamic imports of unknown packages at runtime.
- **Inversion of control**: Markform defines the adapter shape; the client satisfies it. Markform does not need to know about every provider.
- **Backward compatible**: The 5 built-in providers are preserved as defaults. Existing code using `anthropic/claude-sonnet-4-5` works unchanged.
- **AI SDK ergonomic**: Since AI SDK providers are callable, they can be passed directly without wrapping -- `registerProvider('deepinfra', createDeepInfra({ apiKey }))` is a one-liner.
- **Parallel-safe**: Each parallel agent can reconstruct the model from the registered adapter rather than serializing a LanguageModel instance.

## Files Affected

- `src/harness/modelResolver.ts` -- Add registry, make `parseModelId()` accept unknown providers when adapter is registered
- `src/harness/harnessTypes.ts` -- Add `ProviderAdapter`, `ProviderInput` types; add `providers?` to `FillOptions`; widen `ProviderName` or make it extensible
- `src/harness/programmaticFill.ts` -- Update resolution logic in `fillForm()` to check registry
- `src/harness/liveAgent.ts` -- Use adapter `.tools` instead of hard-coded `loadWebSearchTools()` switch
- `src/llms.ts` -- `WEB_SEARCH_CONFIG` and `SUGGESTED_LLMS` could be extended via registration
- `src/index.ts` -- Export `registerProvider`, `ProviderAdapter`, etc.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.