posit-dev / posit-dev/ai-lib

Native Gemini routing for arbitrarily-named external endpoints: carry a profile key through chat params

Open
#63 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
5
Forks
2
Avg merge
13h 33m
Merged PRs (30d)
35

Description

Follow-up to #61 (Databricks native APIs), which defers this.

Problem

Databricks native Gemini routing (google-generative protocol stamp) is restricted to endpoints whose name positively identifies the thinking variant (hosted pay-per-token names like databricks-gemini-2-5-pro). This is because ModelClientChatParams carries only model + thinkingEffort — at chat time, GeminiGenerateContentClient must re-derive the variant profile (budget vs level thinking control, ranges, off-ability) from the model string via ai-config's getGeminiGenerateContentProfile.

An external Gemini endpoint named e.g. my-gemini-endpoint backed by gemini-2.5-flash is classified correctly at discovery (the classifier sees external_model.name), but the client would have no way to reconstruct the variant from the endpoint name, so discovery deliberately stamps openai-chat instead (the classifier and client share one helper so stamp and wire choice cannot disagree).

Proposed direction

Widen the chat-params seam so the catalog pipeline can carry a discovery-time profile key (e.g. the resolved underlying model identity or a variant/profile id) into ModelClientChatParams. The client would prefer the carried key and fall back to name-derivation. Cross-repo change: bridge type plus catalog pipeline plumbing in consumers.

Scope notes

  • Only worth doing if external Gemini endpoints on Databricks (or similar gateways) matter to users; hosted pay-per-token Gemini already routes natively.
  • Could share the seam proposed in #62 (per-protocol capability provenance).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ModelClientChatParams, GeminiGenerateContentClient, and ai-config's getGeminiGenerateContentProfile. Trace the catalog classifier and consumer pipeline to identify where the discovery-time profile key can cross the bridge type. Done means external Gemini endpoints can retain native routing while name-based derivation remains the fallback, with the related cross-repo plumbing covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.