HarperFast / HarperFast/harper

models.embed(): accept image/binary inputs and route to multimodal backend

Open
#765 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Feature Summary

`scope.models.embed()` (and the top-level `models` once #764 lands) is text-only today — `input: string | string[]`. Multimodal embedding backends (CLIP, SigLIP, jina-clip, nomic-vision, etc.) already work through vLLM's OpenAI-compatible \`/v1/embeddings\` shim, but only if user code bypasses Harper's models API and calls the HTTP endpoint directly with the chat-style \`messages: [{ content: [{ type: 'image_url', image_url: { url: 'data:image/jpeg;base64,...' } }] }]\` payload.

## Problem this solves

When I built the harper-celebrity-match demo (selfie → top-5 lookalikes via CLIP), I had to drop \`scope.models.embed()\` entirely and write a per-component \`lib/embed.js\` that reads \`server.config.models.embedding.multimodal\` directly and POSTs to vLLM. That defeats the abstraction — the whole point of \`scope.models\` is that user code doesn't have to know about backend URLs or apiKeys, and we lose:

- Analytics rows in \`hdb_model_calls\` (per-tenant token/GPU billing visibility)
- ALS-bound abort propagation
- Pluggable backend selection by logical model name (\`'multimodal'\` instead of hard-coded URL)
- Per-tenant rate limits (when they ship)

## Proposed Solution

Extend the \`embed\` signature to accept binary or data-URL inputs in addition to strings:

\`\`\`ts
embed(input: string | string[] | Buffer | Uint8Array | ImageInput | (string | ImageInput)[], opts: EmbedOpts): Promise

type ImageInput =
| { type: 'image_url', url: string } // remote URL or data: URL
| { type: 'image_bytes', bytes: Uint8Array, mimeType?: string }
\`\`\`

\`OpenAIBackend.embed()\` (and any other backend that supports image embedding) detects non-string inputs, builds the chat-shape \`messages[].content[]\` payload, and POSTs to the configured \`baseUrl\`.

Backends that don't support image inputs throw \`ModelCapabilityError\` with capability \`'embedImage'\` (or similar) so the existing capability machinery handles routing-by-fallback cleanly.

## User Story

\`\`\`js
import { models, tables } from 'harperdb'

const queryVec = await models.embed(
{ type: 'image_bytes', bytes: req.body, mimeType: 'image/jpeg' },
{ model: 'multimodal' }
)

const matches = tables.Celebrity.search({
conditions: { attribute: 'embedding', comparator: 'lt', value: 2, target: queryVec[0] },
limit: 10,
})
\`\`\`

## Alternatives Considered

- A separate \`models.embedImage(...)\` method. Cleaner in some ways, but every other openai-shaped backend treats embedding as one polymorphic call, and \`models.embed\` is already what users reach for first.
- Pre-serializing to base64 in user code and passing a special string format. Works but loses type information at the API boundary.

## Priority/Impact

Medium — multimodal demos are achievable today by going around \`scope.models\`, but every project that does so duplicates the same backend dispatch logic (and never gets billing visibility).

## Additional Context

See https://github.com/HarperFast/harper-celebrity-match/blob/main/lib/embed.js for the workaround I wrote. It's ~40 lines of code that would be unnecessary if \`models.embed\` accepted binary inputs.

## Are you planning to fix this issue?

I'm open to taking it but want to align on the API shape first.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.