googleapis / googleapis/python-aiplatform

Vertex AI Experiments UI won't show params if using Provisioned Throughput

Offen
#5,085 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
api: vertex-ai type: feature request
Vorherrschende Sprache
Python
Sterne
905
Forks
465
Ø Merge
1 T. 13 Std.
Gemergte PRs (30 T.)
44

Beschreibung

When using a `Callable[[str], str]]]` type for the `model` of `EvalTask.evaluate` no params there isn't a way to supply the parameters used and subsequently, have them displayed on the Vertex AI Experiments UI on the Google Cloud Console.

🚩 **This is particularly problematic for customers using Provision Throughput and adopting the `genai` sdk as per the examples.**

Eg: This displays params

```
model = GenerativeModel(
"gemini-2.0-flash",
generation_config={"temperature": 0.6, "max_output_tokens": 256, "top_k": 1},

)
...
rouge_result = rouge_eval_task.evaluate(
model=model,
prompt_template="# System_prompt\n{system_prompt} # Question\n{question}",
)
```
![Image](https://github.com/user-attachments/assets/55be6543-5dde-41b1-8333-5e52928160d3)

However, if you adapt the `genai` PT example, there is no way to set the params for `EvalTask`
```
client = genai.Client(
http_options=HttpOptions(
...

def use_genai_sdk(prompt: str) -> str:
response = client.models.generate_content(
model="gemini-2.0-flash-001",
contents=prompt,
config={
"temperature": 0.9,
}
)
return response.text

rouge_result = rouge_eval_task.evaluate(
model=use_genai_sdk,
prompt_template="# System_prompt\n{system_prompt} # Question\n{question}",
)
```
And therefore, the Vertex AI Experiments UI is less valuable:

![Image](https://github.com/user-attachments/assets/d4ebb8fe-3e19-4b48-a82f-fbd2269757b0)

https://github.com/googleapis/python-aiplatform/blob/43ebe9793d8dd5a96c59111f93c122413b3e9030/vertexai/evaluation/eval_task.py#L523

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.