spring-projects / spring-projects/spring-ai

Supporting multiple LLMs in auto-config

Open
#2,610 7 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

Currently, proprietary or open models running in Google Vertex can be accessed through the Vertex Gemini module, respectively using the OpenAI API.

#################################
# Google Vertex AI Gemini
#################################
spring.ai.vertex.ai.gemini.project-id=<project>
spring.ai.vertex.ai.gemini.location=us-central1
spring.ai.vertex.ai.gemini.chat.options.model=gemini-2.0-flash-001
spring.ai.vertex.ai.gemini.transport=grpc

#################################
# OpenAI API VertexAI
#################################
spring.ai.openai.api-key=abc123
spring.ai.openai.vertex.ai.gemini.project-id=<project>
spring.ai.openai.vertex.ai.gemini.location=us-central1
spring.ai.openai.vertex.ai.chat.options.model=meta/llama3-405b-instruct-maas
spring.ai.openai.vertex.ai.chat.base-url=<baseURL>
spring.ai.openai.vertex.ai.chat.completions-path=/chat/completions
spring.ai.openai.vertex.ai.chat.options.max-tokens=1024

The problem - there is no ability to auto-config more than 1 Gemini or open model in Vertex at this time.

The proposal - introduce the ability to add a in the auto-config, to allow for multiple configurations

spring.ai.vertex.ai.gemini.<gemini-flash>.project-id=<project>
spring.ai.vertex.ai.gemini.<gemini-flash>.location=us-central1
spring.ai.vertex.ai.gemini.<gemini-flash>.chat.options.model=gemini-2.0-flash-001
spring.ai.vertex.ai.gemini.<gemini-flash>.transport=grpc

spring.ai.vertex.ai.gemini.<gemini-pro>.project-id=<project>
spring.ai.vertex.ai.gemini.<gemini-pro>.location=us-central1
spring.ai.vertex.ai.gemini.<gemini-pro>.chat.options.model=gemini-1.5-001
spring.ai.vertex.ai.gemini.<gemini-pro>.transport=grpc

Workaround - not using auto-config and manually configuring each model in use

CC @tzolov

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Vertex Gemini module and the OpenAI API VertexAI auto-configuration described in the issue. Determine how the current single-model properties are bound and instantiated, then verify that separate model-family properties can create multiple configurations without breaking existing configuration behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java, spring-boot
Domain
ai, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.