HarperFast / HarperFast/harper
Lazy-pull embedding/generative models on first use
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Problem
If the configured embedding model isn't already pulled in ollama, the first `scope.models.embed()` call returns a 404 from ollama with a message like `model 'nomic-embed-text' not found, try pulling it first`. The developer has to break out of their dev loop, `ollama pull `, and retry.
## Proposal
The ollama backend (and any other backend that supports an on-demand pull API) should:
1. Cache, per model name, whether we've successfully called it before.
2. On first call for a model, try the embed request. If the backend returns the canonical \"model-not-found\" error:
- Send `POST /api/pull { name: , stream: false }` to ollama
- Wait for it to complete (can take a few seconds to minutes depending on model size)
- Retry the original embed/generate request once
- On success, mark cached. On failure, throw the original error.
3. Log at info level: `Auto-pulled embedding model 'nomic-embed-text' (took 18.3s, 274 MB)`
## Guardrails
- Only auto-pull when `hm.models.autoPull` is true (default `true` in dev mode, `false` on Fabric — Fabric hosts pre-load via `setupInferenceServer` per the GPU-host plan).
- Single-flight: if 100 concurrent requests arrive while the pull is in flight, only one issues the pull; the rest await.
- Bounded retry — never auto-pull more than once per model per process lifetime, so a model that genuinely doesn't exist on the registry fails fast on subsequent calls.
## Why not just document \"pull before you embed()\"?
Because every dev hits this exactly once and it's pure friction. The auto-pull turns \"copy-paste the example, get an error, look up the fix\" into \"copy-paste the example, wait 18 seconds, see it work.\"
## Related
- PR #779 (Phase 1 model analytics emission)
Contributor guide
Assessment
This issue has not been assessed yet.