HarperFast / HarperFast/harper

Lazy-pull embedding/generative models on first use

Open
#784 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Problem
If the configured embedding model isn't already pulled in ollama, the first `scope.models.embed()` call returns a 404 from ollama with a message like `model 'nomic-embed-text' not found, try pulling it first`. The developer has to break out of their dev loop, `ollama pull `, and retry.

## Proposal
The ollama backend (and any other backend that supports an on-demand pull API) should:

1. Cache, per model name, whether we've successfully called it before.
2. On first call for a model, try the embed request. If the backend returns the canonical \"model-not-found\" error:
- Send `POST /api/pull { name: , stream: false }` to ollama
- Wait for it to complete (can take a few seconds to minutes depending on model size)
- Retry the original embed/generate request once
- On success, mark cached. On failure, throw the original error.
3. Log at info level: `Auto-pulled embedding model 'nomic-embed-text' (took 18.3s, 274 MB)`

## Guardrails
- Only auto-pull when `hm.models.autoPull` is true (default `true` in dev mode, `false` on Fabric — Fabric hosts pre-load via `setupInferenceServer` per the GPU-host plan).
- Single-flight: if 100 concurrent requests arrive while the pull is in flight, only one issues the pull; the rest await.
- Bounded retry — never auto-pull more than once per model per process lifetime, so a model that genuinely doesn't exist on the registry fails fast on subsequent calls.

## Why not just document \"pull before you embed()\"?
Because every dev hits this exactly once and it's pure friction. The auto-pull turns \"copy-paste the example, get an error, look up the fix\" into \"copy-paste the example, wait 18 seconds, see it work.\"

## Related
- PR #779 (Phase 1 model analytics emission)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.