Menci / Menci/Floway

chat/completions: streamed logprobs are dropped during reassembly — only last chunk survives

Open
#481 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
85
Forks
33
Avg merge
45m
Merged PRs (30d)
10

Description

Summary

When a client requests logprobs: true with stream: true, the non-streaming result (and the SSE stream) only contains the logprobs of the last output token. All intermediate token logprobs are lost.

Repro

Request (official DeepSeek API as upstream):

POST /v1/chat/completions
{ "model": "deepseek-v4-flash", "messages": [{ "role": "user", "content": "Reply with exactly: alpha beta gamma delta" }], "thinking": {"type":"disabled"}, "logprobs": true, "top_logprobs": 3, "max_tokens": 100, "stream": true }

Direct to upstream: 6 chunks, 4 content tokens, all 4 carry logprobs ✅
Through Floway: 3 chunks, only the final chunk carries logprobs (1-2 tokens) ❌

Same with non-streaming: Floway returns logprobs.content with 1 token instead of 4.

Root cause

packages/protocols/src/chat-completions/reassemble.ts:

  • KNOWN_DELTA_KEYS does not include logprobs → it falls through to captureExtras
  • captureExtras uses last-write-wins for non-string non-array values → each chunk overwrites the previous logprobs, leaving only the final chunk's

Same pattern exists in packages/protocols/src/completions/reassemble.ts (accumulator.logprobs = choice.logprobs overwrite).

Expected

Logprobs accumulate across chunks (append to the content array), so logprobs.content has one entry per output token, matching the non-streaming response. OpenAI's streaming wire format sends logprobs per-token in each chunk's delta.logprobs.content — they should be concatenated, not replaced.

Environment

Floway via pnpm workspace monorepo (gateway + protocols packages), upstream = DeepSeek official API (api.deepseek.com, chatCompletions endpoint).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/protocols/src/chat-completions/reassemble.ts, tracing KNOWN_DELTA_KEYS and captureExtras, then compare the overwrite in packages/protocols/src/completions/reassemble.ts. Use the supplied streaming and non-streaming DeepSeek reproductions to verify that logprobs.content accumulates one entry per output token rather than retaining only the final chunk.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend-api-design
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.