anthropics / anthropics/anthropic-sdk-python
Bedrock: support low-latency configuration
- Ngôn ngữ chính
- Python
- Star
- 3.9k
- Fork
- 853
- Merge trung bình
- 1 ngày 11 giờ
- Pull request đã merge (30 ngày)
- 10
Mô tả
Currently, `[Async]AnthropicBedrock` doesn't provide a way to enable low-latency mode.
It also appears that this configuration isn't even accessible via the Bedrock Anthropic HTTP API.
So the only option for the Antropic clients to use this low-latency model is to use the Converse API provided by AWS Becrock: https://docs.aws.amazon.com/bedrock/latest/userguide/latency-optimized-inference.html
This is very inconvenient, since the Converse API unifies APIs for multiple models and, therefore, it lacks support for exclusive features of particular models _(e.g. `tool_choice=none` is missing from the Converse API)_.
Exposing low-latency configuration in `[Async]AnthropicBedrock` would be great.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.