aws / aws/aws-sdk-ruby

BedrockAgentCore: invoke_agent_runtime buffers entire response instead of streaming in real-time

Open
#3,348 6 comments 1 reaction 0 assignees View on GitHub
documentation
Dominant language
Ruby
Stars
3.7k
Forks
1.2k
Avg merge
1d 4h
Merged PRs (30d)
5

Description

### Describe the bug

The `invoke_agent_runtime` method buffers the entire EventStream response before yielding any chunks to `resp.response.each`, completely defeating the purpose of streaming. All chunks arrive simultaneously after the full response is generated, making real-time streaming UX impossible.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

Chunks should arrive progressively as the AWS agent generates them, similar to BedrockRuntime's `invoke_model_with_response_stream`:

- Chunk 1 arrives at ~150ms
- Chunk 2 arrives at ~450ms
- Chunk 3 arrives at ~1200ms
- etc.

This enables real-time UX like typing indicators and progressive responses in chatbot applications.

### Current Behavior

All chunks arrive within 1-2ms after waiting for the complete response:

D, [2026-01-21T01:15:34.902128] [AwsBedrockService] Chunk 1 at 25520.21ms, length=31
D, [2026-01-21T01:15:34.902602] [AwsBedrockService] Chunk 3 at 25520.78ms, length=21
D, [2026-01-21T01:15:34.902683] [AwsBedrockService] Chunk 5 at 25520.87ms, length=32
D, [2026-01-21T01:15:34.902734] [AwsBedrockService] Chunk 7 at 25520.92ms, length=56

All 4 chunks delivered within 0.75ms at the 25.5-second mark, despite the agent taking 25+ seconds to generate the response. The SDK is clearly buffering internally.

### Reproduction Steps

```
require 'aws-sdk-bedrockagentcore'
require 'securerandom'
require 'json'

# This reproduces buffering issue with BedrockAgentCore streaming
client = Aws::BedrockAgentCore::Client.new(region: 'us-west-2')

# NOTE: Replace with your actual agent runtime ARN
# Example format: arn:aws:bedrock-agentcore:us-west-2:123456789012:runtime/abc123
agent_arn = ENV.fetch('AGENT_RUNTIME_ARN')

start_time = Time.now
puts "Starting request at #{Time.now}"

resp = client.invoke_agent_runtime(
agent_runtime_arn: agent_arn,
runtime_session_id: SecureRandom.uuid,
payload: JSON.generate({ input: { prompt: 'Tell me a long story' } })
)

puts "\nIterating over response stream:"
chunk_count = 0

resp.response.each do |chunk|
chunk_count += 1
elapsed_ms = ((Time.now - start_time) * 1000).round(2)
puts "Chunk ##{chunk_count} at #{elapsed_ms}ms: #{chunk.bytesize} bytes"
end

puts "\nExpected: Chunks arriving progressively (e.g., 100ms, 500ms, 1000ms)"
puts "Actual: All chunks arriving at approximately the same time (buffering)"

```
Expected: Chunks arriving progressively (100ms, 500ms, 1000ms, etc.)
Actual: All chunks arrive at once after full response completes

### Possible Solution

Add event_stream_handler support similar to aws-sdk-bedrockruntime's invoke_model_with_response_stream:

```
client.invoke_agent_runtime(
agent_runtime_arn: '...',
payload: '...',
event_stream_handler: proc { |stream|
stream.on_event { |event| puts event }
}
)
```

Alternatively, fix resp.response.each to yield chunks as they arrive from the HTTP connection rather than buffering the entire response first.

We've confirmed that raw Net::HTTP with AWS SigV4 signing streams correctly in real-time, proving the service supports it but the SDK buffers.

### Additional Information/Context

This issue makes aws-sdk-bedrockagentcore unusable for real-time applications requiring streaming UX (chatbots, live agents, etc.).

Workaround: We've implemented a monkey patch using raw Net::HTTP that successfully streams in real-time by bypassing the SDK's invoke_agent_runtime method entirely and making direct HTTP requests with streaming enabled.

The BedrockRuntime service gem (aws-sdk-bedrockruntime) has proper event streaming support via event_stream_handler, but BedrockAgentCore lacks this functionality.

### Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-bedrockagentcore version 1.14.0

### Environment details (Version of Ruby, OS environment)

- Ruby version: 3.2.6 - Operating System: macOS (Darwin 23.1.0) - Rails: 8.0.2.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.