HarperFast / HarperFast/harper

models: generation backends drop provider options (Ollama num_ctx) → silent prompt truncation

Open
#1,775 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

Generation/embedding backends drop provider-specific options — notably **Ollama `num_ctx`** — so long prompts are **silently truncated** to the backend default. There's no way to pass these options through `models.generate` / `models.embed`.

## What happens

`components/ollama/index.ts` `buildOptionsBag` forwards only `temperature` and `num_predict` (from `maxTokens`). It never sets `num_ctx`. Ollama's default context window is **2048 tokens**, so any prompt longer than that is quietly clipped — and crucially this presents as **low-quality output, not an error**: the model returns a plausible-looking but degraded response (e.g. meta-language like *"Based on the provided code snippet…"* instead of a real summary).

I hit this building the RAPTOR summary-tree component (#1244 Thread B). Large-cluster summaries were consistently garbage while small ones were fine — because only the large clusters overran 2048 tokens. It read as "the 3B model is dumb" until I checked context length. The only workaround was baking `num_ctx` into a derived model (`ollama create … -f Modelfile` with `PARAMETER num_ctx 8192`), which a component author shouldn't have to do.

## Ask

A pass-through for backend options on the `generate`/`embed` opts, at minimum `num_ctx` (or a backend-agnostic `contextWindow`) for Ollama. This generalizes: every backend has provider-specific knobs (OpenAI/Azure params, watsonx params, etc.) that currently have no route through the facade. Options:

- a typed `contextWindow` (mapped per-backend: Ollama `num_ctx`, others where applicable), and/or
- an escape-hatch `providerOptions: { … }` bag merged into the backend request.

Whatever the shape, the truncation should also **not be silent** — a prompt that exceeds the configured/known context window is worth a warning.

## Notes

- Sibling to #1757 (models facade lacks a metadata accessor) — both are "the facade hides something a serious consumer needs."
- Filed from the #1244 friction list.

🤖 Filed by Claude (Opus 4.8) on behalf of Kris.

Contributor guide

Open the contributing guide

Research direction

Start in components/ollama/index.ts at buildOptionsBag, then trace the models.generate and models.embed option types and backend request path. Define how provider options or a context window pass through, ensure Ollama receives num_ctx, and verify that prompts exceeding the configured or known window are no longer silently truncated.

Written by the indexing model from the issue text.

Assessment

Tech stack
ollama, typescript
Domain
ai, backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.