google / google/adk-python

[QUESTION] AgentEvaluator.evaluate_eval_set

Ouverte
#1,792 2 commentaires 0 réactions 1 personne assignée Réclamée par @ankursharmas Voir sur GitHub
eval needs review
Langage dominant
Python
Étoiles
21.5k
Forks
4k
Merge moyen
1 j 14 h
PR mergées (30 j)
37

Description

# Description:

We're building an HTTP endpoint that uses AgentEvaluator from ADK to evaluate an agent on-demand. The challenge we're facing is that the evaluator currently requires an agent_module (i.e., a Python module path), and not a preconstructed agent instance.

In our case, we want to initialize the agent with runtime-specific parameters — for example, an account_id extracted from an HTTP header. This value is dynamic and cannot be hardcoded in the agent module, nor is it available at module load time.

However, since AgentEvaluator.evaluate(...) only accepts the module path, we're unable to pass this context into the agent. The agent has to be constructed as the "root agent" in a fixed folder structure.

What we need / feature request:

It would be very helpful if AgentEvaluator supported an additional option to provide an already-initialized agent instance, like:

```python
agent = MyAgent(account_id="abc-123")
AgentEvaluator.evaluate_eval_set(agent=agent, ...)
```
### Workarounds we've tried:

Dynamic agent module generation: We’ve also tried programmatically generating the agent directory structure (that constructs the agent with injected values), then importing and using that as the agent_module. This technically works, but it's weird.

# Question

Is there currently any recommended way to achieve this kind of dynamic initialization for evaluation use cases like this?
If there's an alternative pattern that would let us pass info (like account_id) into the agent at runtime while still using AgentEvaluator, we’d love to hear it.

Thanks a lot in advance! 🙏

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.