posit-dev / posit-dev/chatlas

Support configuring context window size (`num_ctx`) for local models in `ChatOllama`

Open
#327 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ai-triage:done Priority: Medium
Dominant language
Python
Stars
176
Forks
28
Avg merge
18h 42m
Merged PRs (30d)
16

Description

When using ChatOllama in chatlas to talk to local models (especially models with reasoning capabilities like gemma4 or larger parameter models), responses are easily truncated due to Ollama's default context window limit of 2048 tokens.

Because chatlas (seemingly) interfaces with Ollama via the OpenAI-compatible /v1/chat/completions API and strictly filters parameters in translate_model_params(), there is currently no way for users to increase the context window size (num_ctx) on-the-fly.

In contrast, the R equivalent (ellmer::chat_ollama) handles this seamlessly by communicating directly with Ollama's native /api/chat API, which natively supports setting num_ctx and num_predict via the request options.


Reproducible Examples (Reprex)

1. R (ellmer) - Success

Using ellmer, we can configure both max output tokens and a large context window, avoiding truncation:

library(ellmer)

# ellmer maps these parameters directly to Ollama's options block
chat <- chat_ollama(
  model = "gemma4",
  params = params(
    max_tokens = 2048,
    num_ctx = 8192
  )
)

response <- chat$chat("Write a very long essay about the history of statistics.")
cat(nchar(response)) # Successfully generates a long, complete response
2. Python (chatlas) - Truncation (Failure)

Using chatlas, setting max_tokens is supported, but we cannot pass num_ctx to the client. When the prompt + response (including thinking traces) exceeds 2048 tokens, it gets truncated:

from chatlas import ChatOllama

# Initialize client (no way to configure context window)
chat = ChatOllama(model="gemma4")
chat.set_model_params(max_tokens=2048)

# Writing a long prompt/response will hit Ollama's default 2048 context limit
response = chat.chat("Write a very long essay about the history of statistics.")
print(chat.get_turns()[-1].finish_reason) # prints "length" (truncated early)

Proposed Solutions

  1. Allow custom parameters in ChatOllama:
    Expose num_ctx (or a generic options payload dictionary) in the ChatOllama client initialization, and have the underlying OllamaProvider pass it along as part of the request payload to Ollama's completion endpoint (Ollama's /v1/chat/completions accepts custom parameters like num_ctx in the JSON request body).

  2. Expose provider-specific parameters in set_model_params():
    Instead of raising a TypeError for non-standard parameters, allow provider-specific keys like num_ctx to be passed through to the underlying request payload.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing ChatOllama through translate_model_params() into OllamaProvider and review how set_model_params() filters arguments for the /v1/chat/completions payload. Compare the proposed num_ctx or options handling with Ollama's supported request fields. Done means users can configure num_ctx for local models without a TypeError and the value reaches the request payload.

Written by the indexing model from the issue text.

Assessment

Tech stack
ollama, python
Domain
ai, api
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.