googleapis / googleapis/python-aiplatform

Vertex AI Experiments UI won't show params if using Provisioned Throughput

Aperta
#5,085 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
api: vertex-ai type: feature request
Lingua principale
Python
Stelle
905
Fork
465
Merge medio
1g 13h
PR unite (30g)
44

Descrizione

When using a `Callable[[str], str]]]` type for the `model` of `EvalTask.evaluate` no params there isn't a way to supply the parameters used and subsequently, have them displayed on the Vertex AI Experiments UI on the Google Cloud Console.

🚩 **This is particularly problematic for customers using Provision Throughput and adopting the `genai` sdk as per the examples.**

Eg: This displays params

```
model = GenerativeModel(
"gemini-2.0-flash",
generation_config={"temperature": 0.6, "max_output_tokens": 256, "top_k": 1},

)
...
rouge_result = rouge_eval_task.evaluate(
model=model,
prompt_template="# System_prompt\n{system_prompt} # Question\n{question}",
)
```
![Image](https://github.com/user-attachments/assets/55be6543-5dde-41b1-8333-5e52928160d3)

However, if you adapt the `genai` PT example, there is no way to set the params for `EvalTask`
```
client = genai.Client(
http_options=HttpOptions(
...

def use_genai_sdk(prompt: str) -> str:
response = client.models.generate_content(
model="gemini-2.0-flash-001",
contents=prompt,
config={
"temperature": 0.9,
}
)
return response.text

rouge_result = rouge_eval_task.evaluate(
model=use_genai_sdk,
prompt_template="# System_prompt\n{system_prompt} # Question\n{question}",
)
```
And therefore, the Vertex AI Experiments UI is less valuable:

![Image](https://github.com/user-attachments/assets/d4ebb8fe-3e19-4b48-a82f-fbd2269757b0)

https://github.com/googleapis/python-aiplatform/blob/43ebe9793d8dd5a96c59111f93c122413b3e9030/vertexai/evaluation/eval_task.py#L523

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.