googleapis / googleapis/python-aiplatform

Vertex AI Experiments UI won't show params if using Provisioned Throughput

Open
#5,085 0 comments 0 reactions 0 assignees View on GitHub
api: vertex-ai type: feature request
Dominant language
Python
Stars
905
Forks
465
Avg merge
1d 13h
Merged PRs (30d)
44

Description

When using a `Callable[[str], str]]]` type for the `model` of `EvalTask.evaluate` no params there isn't a way to supply the parameters used and subsequently, have them displayed on the Vertex AI Experiments UI on the Google Cloud Console.

🚩 **This is particularly problematic for customers using Provision Throughput and adopting the `genai` sdk as per the examples.**

Eg: This displays params

```
model = GenerativeModel(
"gemini-2.0-flash",
generation_config={"temperature": 0.6, "max_output_tokens": 256, "top_k": 1},

)
...
rouge_result = rouge_eval_task.evaluate(
model=model,
prompt_template="# System_prompt\n{system_prompt} # Question\n{question}",
)
```
![Image](https://github.com/user-attachments/assets/55be6543-5dde-41b1-8333-5e52928160d3)

However, if you adapt the `genai` PT example, there is no way to set the params for `EvalTask`
```
client = genai.Client(
http_options=HttpOptions(
...

def use_genai_sdk(prompt: str) -> str:
response = client.models.generate_content(
model="gemini-2.0-flash-001",
contents=prompt,
config={
"temperature": 0.9,
}
)
return response.text

rouge_result = rouge_eval_task.evaluate(
model=use_genai_sdk,
prompt_template="# System_prompt\n{system_prompt} # Question\n{question}",
)
```
And therefore, the Vertex AI Experiments UI is less valuable:

![Image](https://github.com/user-attachments/assets/d4ebb8fe-3e19-4b48-a82f-fbd2269757b0)

https://github.com/googleapis/python-aiplatform/blob/43ebe9793d8dd5a96c59111f93c122413b3e9030/vertexai/evaluation/eval_task.py#L523

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.