googleapis / googleapis/python-aiplatform

Question about Vertex AI EvalTask Judge Model

Offen
#6,042 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
api: vertex-ai
Vorherrschende Sprache
Python
Sterne
905
Forks
465
Ø Merge
1 T. 13 Std.
Gemergte PRs (30 T.)
44

Beschreibung

Hi, I am using the Vertex AI Gen AI Evaluation SDK, specifically the EvalTask for evaluating model responses. I see that the judge model for evaluation is typically a Vertex AI hosted Gemini model, and you can customize it by tuning a Gemini model using the SDK. (https://cloud.google.com/vertex-ai/generative-ai/docs/models/configure-judge-model )

My question is:
Is it possible to replace the judge model with an external model served outside of Vertex AI, such as a custom REST API endpoint (for example, a Databricks serving endpoint or any other external model API)? In other words, can the judge model in EvalTask be configured to call an external model API instead of a Vertex AI managed model?

If not supported, are there any recommended ways or workarounds to integrate external judge models into the Vertex AI evaluation workflow?

Thank you very much for any guidance!

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.