posit-dev / posit-dev/chatlas

A streaming text delta that repeats everything streamed so far is missing from the recorded turn

Open Beginner friendly
#435 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ai-triage:done Priority: Medium
Dominant language
Python
Stars
176
Forks
28
Avg merge
18h 42m
Merged PRs (30d)
16

Description

If a streamed response contains a text delta identical to everything streamed before it, that delta does not make it onto the assistant turn. The caller watches the right text go by, and the turn that gets stored, and sent back as history on the next request, is short by one delta.

The easy case is a response starting with two identical deltas, which is what a leading "\n\n" often looks like coming off a local model.

No API key needed for this, it drives the completions provider's streaming contract:

from openai.types.chat import ChatCompletionChunk
from chatlas._provider_openai_completions import OpenAICompletionsProvider

provider = OpenAICompletionsProvider(api_key="unused", model="gpt-4o-mini")

def chunk(text, last=False):
    return ChatCompletionChunk.model_validate({
        "id": "chatcmpl-1",
        "object": "chat.completion.chunk",
        "created": 1,
        "model": "gpt-4o-mini",
        "choices": [{
            "index": 0,
            "delta": {"role": "assistant", "content": text},
            "finish_reason": "stop" if last else None,
        }],
    })

deltas = ["\n", "\n", "Here is the answer."]

completion = None
for i, d in enumerate(deltas):
    completion = provider.stream_merge_chunks(completion, chunk(d, last=i == len(deltas) - 1))
turn = provider.stream_turn(completion, has_data_model=False)

print("streamed:", repr("".join(deltas)))
print("turn:    ", repr(turn.text))
streamed: '\n\nHere is the answer.'
turn:     '\nHere is the answer.'

I expected both to be '\n\nHere is the answer.'.

It is not just the first two deltas. ["ab", "c", "abc"] streams abcabc and records abc, while ["ab", "c", "ab"] comes out fine, so it seems to bite when the new delta equals the whole accumulation so far.

chatlas at 7b3aa569, openai 3.13.0, python 3.12.14. Everything built on OpenAICompletionsProvider shares that call, so ChatOllama(), ChatGroq(), ChatDeepSeek() and ChatOpenRouter() are in the same boat. I found it driving the provider rather than a live model, so I cannot say how often a real stream lands on it.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with OpenAICompletionsProvider.stream_merge_chunks and stream_turn, using the supplied ChatCompletionChunk reproduction to trace how repeated deltas are accumulated and recorded. Add a regression test covering ["\n", "\n", "Here is the answer."] and the other examples, then verify that the recorded turn matches the complete streamed text.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.