anthropics / anthropics/anthropic-sdk-python

Bedrock: support low-latency configuration

Abierto
#971 0 comentarios 0 reacciones 0 asignados Ver en GitHub
bedrock
Lenguaje dominante
Python
Estrellas
3.9k
Forks
853
Merge medio
1 d 11 h
PR fusionados (30 d)
10

Descripción

Currently, `[Async]AnthropicBedrock` doesn't provide a way to enable low-latency mode.
It also appears that this configuration isn't even accessible via the Bedrock Anthropic HTTP API.

So the only option for the Antropic clients to use this low-latency model is to use the Converse API provided by AWS Becrock: https://docs.aws.amazon.com/bedrock/latest/userguide/latency-optimized-inference.html

This is very inconvenient, since the Converse API unifies APIs for multiple models and, therefore, it lacks support for exclusive features of particular models _(e.g. `tool_choice=none` is missing from the Converse API)_.

Exposing low-latency configuration in `[Async]AnthropicBedrock` would be great.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.