HarperFast / HarperFast/harper
models: generation backends drop provider options (Ollama num_ctx) → silent prompt truncation
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
Generation/embedding backends drop provider-specific options — notably **Ollama `num_ctx`** — so long prompts are **silently truncated** to the backend default. There's no way to pass these options through `models.generate` / `models.embed`.
## What happens
`components/ollama/index.ts` `buildOptionsBag` forwards only `temperature` and `num_predict` (from `maxTokens`). It never sets `num_ctx`. Ollama's default context window is **2048 tokens**, so any prompt longer than that is quietly clipped — and crucially this presents as **low-quality output, not an error**: the model returns a plausible-looking but degraded response (e.g. meta-language like *"Based on the provided code snippet…"* instead of a real summary).
I hit this building the RAPTOR summary-tree component (#1244 Thread B). Large-cluster summaries were consistently garbage while small ones were fine — because only the large clusters overran 2048 tokens. It read as "the 3B model is dumb" until I checked context length. The only workaround was baking `num_ctx` into a derived model (`ollama create … -f Modelfile` with `PARAMETER num_ctx 8192`), which a component author shouldn't have to do.
## Ask
A pass-through for backend options on the `generate`/`embed` opts, at minimum `num_ctx` (or a backend-agnostic `contextWindow`) for Ollama. This generalizes: every backend has provider-specific knobs (OpenAI/Azure params, watsonx params, etc.) that currently have no route through the facade. Options:
- a typed `contextWindow` (mapped per-backend: Ollama `num_ctx`, others where applicable), and/or
- an escape-hatch `providerOptions: { … }` bag merged into the backend request.
Whatever the shape, the truncation should also **not be silent** — a prompt that exceeds the configured/known context window is worth a warning.
## Notes
- Sibling to #1757 (models facade lacks a metadata accessor) — both are "the facade hides something a serious consumer needs."
- Filed from the #1244 friction list.
🤖 Filed by Claude (Opus 4.8) on behalf of Kris.
Contributor guide
Research direction
Start in components/ollama/index.ts at buildOptionsBag, then trace the models.generate and models.embed option types and backend request path. Define how provider options or a context window pass through, ensure Ollama receives num_ctx, and verify that prompts exceeding the configured or known window are no longer silently truncated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ollama, typescript
- Domain
- ai, backend-api-design
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100