spring-projects / spring-projects/spring-ai

[bedrock] Allow injecting pre-built BedrockRuntimeClient / BedrockRuntimeAsyncClient into *BedrockApi classes (parity with BedrockProxyChatModel)

Open
#6,450 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

Expected Behavior

It would be great if AbstractBedrockApi and its subclasses (e.g. CohereEmbeddingBedrockApi, TitanEmbeddingBedrockApi) allow injecting pre-built BedrockRuntimeClient / BedrockRuntimeAsyncClient instances, the same way BedrockProxyChatModel.Builder already does:

// One API instance per tenant, each wired with its own token-configured client
BedrockRuntimeClient client = BedrockRuntimeClient.builder()
        .region(region)
        .tokenProvider(StaticTokenProvider.create(() -> tenant.getBearerToken()))
        .authSchemeProvider(BedrockRuntimeAuthSchemeProvider
                .defaultProvider(List.of("httpBearerAuth")))
        .build();

var api = CohereEmbeddingBedrockApi.builder()
        .modelId(CohereEmbeddingModel.COHERE_EMBED_MULTILINGUAL_V3.id())
        .bedrockRuntimeClient(client)
        .bedrockRuntimeAsyncClient(asyncClient)
        .build();

Current Behavior

AbstractBedrockApi only exposes constructors that take an AwsCredentialsProvider. The internal clients are private final and built inside the constructor

There is no supported way to supply pre-built clients. This is asymmetric with the chat side: BedrockProxyChatModel.Builder already accepts bedrockRuntimeClient(...) / bedrockRuntimeAsyncClient(...). The embedding APIs (and any other API extending AbstractBedrockApi) cannot do the same.

This blocks any AWS SDK client configuration that cannot be expressed through AwsCredentialsProvider, most notably bearer-token authentication ("API keys for Amazon Bedrock"), which requires:

BedrockRuntimeClient.builder()
        .region(region)
        .tokenProvider(StaticTokenProvider.create(() -> bearerToken))
        .authSchemeProvider(BedrockRuntimeAuthSchemeProvider
                .defaultProvider(List.of("httpBearerAuth")))
        .build();

Context

We're integrating Spring AI Bedrock in a service that authenticates to Bedrock with a bearer token (AWS API key for Bedrock). For the chat model this works out of the box because BedrockProxyChatModel.Builder accepts our token-configured BedrockRuntimeClient directly. For embedding models (Cohere, Titan) we hit the wall described above.

Setting the token via the AWS_BEARER_TOKEN_BEDROCK environment variable (picked up automatically by the AWS SDK's default token provider chain) is not an option for us: the service is multi-tenant and each tenant's configuration carries its own bearer token. We need to construct one BedrockRuntimeClient per token at runtime and route requests through the right one. This is only expressible programmatically by building the client with StaticTokenProvider.create(...) and injecting it into the API class — which is exactly what's missing.

Alternatives / workarounds considered:

  1. Subclass the API class and override internalInvocation / internalInvocationStream. Forces copy-pasting ~60 lines of upstream code per API class with drift risk on every Spring AI release. Also, the superclass constructor still builds two SDK clients (sync + async, with their own connection pools and threads) that the subclass never uses. This is what we currently do:
public class CustomCohereEmbeddingBedrockApi extends CohereEmbeddingBedrockApi {

    private final BedrockRuntimeClient bedrockRuntimeClient;
    private final BedrockRuntimeAsyncClient bedrockRuntimeAsyncClient;
    private final JsonMapper jsonMapper = JacksonUtils.getDefaultJsonMapper();

    public CustomCohereEmbeddingBedrockApi(String modelId, String region,
            BedrockRuntimeClient client, BedrockRuntimeAsyncClient asyncClient) {
        super(modelId, region); // parent builds two unused SDK clients
        this.bedrockRuntimeClient = client;
        this.bedrockRuntimeAsyncClient = asyncClient;
    }

    @Override
    protected CohereEmbeddingResponse internalInvocation(
            CohereEmbeddingRequest request, Class<CohereEmbeddingResponse> clazz) {
        // ~25 lines copied verbatim from AbstractBedrockApi, only the client reference changes
    }

    @Override
    protected Flux<CohereEmbeddingResponse> internalInvocationStream(...) {
        // ~60 more lines copied verbatim (and never executed — embeddings don't stream)
    }
}
  1. Reimplement EmbeddingModel directly, bypassing *BedrockApi entirely. Possible but means giving up the framework's API surface for this provider.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with AbstractBedrockApi and BedrockProxyChatModel.Builder, comparing how each constructs or accepts synchronous and asynchronous BedrockRuntime clients. Inspect the CohereEmbeddingBedrockApi and TitanEmbeddingBedrockApi builders and constructors. Done means embedding APIs can accept pre-built clients while retaining the existing credentials-based path.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend-api-design, cloud
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.