cloudwego / cloudwego/eino-ext

OpenAI ChatModel Timeout interrupts active streaming responses after total duration

Open
#858 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
811
Forks
368
Avg merge
16h 22m
Merged PRs (30d)
13

Description

```md
**Describe the bug**

When using `components/model/openai.ChatModelConfig.Timeout` with streaming enabled, an active streaming response is cancelled after the configured timeout duration, even if chunks are still being received continuously.

The error looks like:

```text
[NodeRunError] failed to receive stream chunk: context deadline exceeded (Client.Timeout or context cancellation while reading body)
------------------------
node path: [ChatModel]
```

From reading the source, `ChatModelConfig.Timeout` is passed to `http.Client{Timeout: config.Timeout}`. In Go, `http.Client.Timeout` covers the entire request lifetime, including reading the response body after `Do` returns. For SSE/chat streaming, this means long-running model output can be interrupted at the timeout boundary.

**To Reproduce**

Steps to reproduce the behavior:

1. Create an OpenAI ChatModel with a finite timeout, for example:

```go
chatModel, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{
APIKey: "...",
Model: "...",
Timeout: 60 * time.Second,
})
```

2. Use the model in streaming mode.
3. Send a prompt that causes the model to stream output for longer than the configured timeout.
4. Observe that the stream is interrupted at around the configured timeout, even while chunks are actively being received.
5. The error is reported as:

```text
failed to receive stream chunk: context deadline exceeded (Client.Timeout or context cancellation while reading body)
```

**Expected behavior**

For streaming responses, an active stream that is receiving chunks should not be cancelled by a total request timeout unless this behavior is explicitly documented.

Expected options could be one of:

- Clarify in the documentation that `ChatModelConfig.Timeout` maps to `http.Client.Timeout` and includes full response body reading.
- Provide separate timeout options, such as:
- response header timeout / first-byte timeout
- stream idle timeout
- non-streaming total request timeout
- Avoid applying `http.Client.Timeout` to streaming requests by default.

**Screenshots**

Not applicable.

**Version:**

`github.com/cloudwego/eino-ext/components/model/openai v0.1.13`

Related dependency observed locally:

`github.com/cloudwego/eino-ext/libs/acl/openai v0.1.17`

**Environment:**

```text
GOVERSION='go1.25.5'
```

**Additional context**

Relevant source behavior:

- `ChatModelConfig.Timeout` is documented as:

```go
// Timeout specifies the maximum duration to wait for API responses
// If HTTPClient is set, Timeout will not be used.
// Optional. Default: no timeout
Timeout time.Duration `json:"timeout"`
```

- `NewChatModel` constructs the default HTTP client as:

```go
httpClient = &http.Client{Timeout: config.Timeout}
```

- Go's `http.Client.Timeout` documentation says the timeout includes connection time, redirects, and reading the response body. The timer remains running after `Do` returns and can interrupt reading `Response.Body`.

This is surprising for chat/SSE streaming because users may expect `Timeout` to mean waiting for the API response to start, not the maximum total duration of a stream that is still producing chunks.
```

Contributor guide

Open the contributing guide

Research direction

Start at components/model/openai.ChatModelConfig.Timeout and the NewChatModel path that constructs http.Client{Timeout: config.Timeout}. Compare the default client behavior with the streaming response path and the reported reproduction. Done requires an agreed timeout policy for streaming, with the relevant behavior documented or covered by verification.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.