googleapis / googleapis/python-aiplatform
Vertex AI Experiments UI won't show params if using Provisioned Throughput
- Vorherrschende Sprache
- Python
- Sterne
- 905
- Forks
- 465
- Ø Merge
- 1 T. 13 Std.
- Gemergte PRs (30 T.)
- 44
Beschreibung
When using a `Callable[[str], str]]]` type for the `model` of `EvalTask.evaluate` no params there isn't a way to supply the parameters used and subsequently, have them displayed on the Vertex AI Experiments UI on the Google Cloud Console.
🚩 **This is particularly problematic for customers using Provision Throughput and adopting the `genai` sdk as per the examples.**
Eg: This displays params
```
model = GenerativeModel(
"gemini-2.0-flash",
generation_config={"temperature": 0.6, "max_output_tokens": 256, "top_k": 1},
)
...
rouge_result = rouge_eval_task.evaluate(
model=model,
prompt_template="# System_prompt\n{system_prompt} # Question\n{question}",
)
```

However, if you adapt the `genai` PT example, there is no way to set the params for `EvalTask`
```
client = genai.Client(
http_options=HttpOptions(
...
def use_genai_sdk(prompt: str) -> str:
response = client.models.generate_content(
model="gemini-2.0-flash-001",
contents=prompt,
config={
"temperature": 0.9,
}
)
return response.text
rouge_result = rouge_eval_task.evaluate(
model=use_genai_sdk,
prompt_template="# System_prompt\n{system_prompt} # Question\n{question}",
)
```
And therefore, the Vertex AI Experiments UI is less valuable:

https://github.com/googleapis/python-aiplatform/blob/43ebe9793d8dd5a96c59111f93c122413b3e9030/vertexai/evaluation/eval_task.py#L523
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.