google / google/adk-python

Align `adk web` eval logic with `adk eval` CLI & `pytest`

Ouverte
#4,867 1 commentaire 2 réactions 2 personnes assignées Réclamée par @Jacksunwei Voir sur GitHub
eval needs review
Langage dominant
Python
Étoiles
21.5k
Forks
4k
Merge moyen
1 j 14 h
PR mergées (30 j)
37

Description

### Is your feature request related to a specific problem?
`adk web` eval discovery is inconsistent with `adk eval` and pytest workflows

There are three ways to run evals in ADK, from what I can tell, and only one of them constrains where eval files must live:

- **`adk eval`** accepts eval set file paths as direct positional arguments — files can be anywhere on disk:
```bash
adk eval my_agent path/to/evals/my_eval.evalset.json
```
- **pytest** (via `AgentEvaluator.evaluate()`) accepts a file path as a parameter — files can be anywhere on disk:
```python
await AgentEvaluator.evaluate("my_agent", "path/to/evals/my_eval.evalset.json")
```
- **`adk web`** uses `LocalEvalSetsManager`, which hardcodes discovery to a flat `os.listdir` on `//` with no way to override it. The only alternative is `--eval_storage_uri=gs://...` for GCS — there is no local path equivalent. Files must sit directly at `//*.evalset.json`.

This means a project following the convention from Google's own [adk-samples](https://github.com/google/adk-samples/tree/main/python/agents) — where eval files live in an `/eval/` subdirectory and are run via pytest or `adk eval` — gets no visibility into those evals from the `adk web` UI. Eval sets created through the `adk web` UI are also written to the agent root via `_get_eval_set_file_path`, diverging from where the project's other eval files are stored.

### Describe the Solution You'd Like
Allow for `adk web` to be configured (through an environment variable or otherwise) with a path where `*.evalset.json` files should be created and read from.

### Impact on your work
Cannot provide a coherent developer experience where JSON files are created through the web UI and then run in CI/CD without polluting the agent app directory with `evalset.json` files.

### Willingness to contribute
Are you interested in implementing this feature yourself or submitting a PR?
Maybe

### Describe Alternatives You've Considered
Using `evalset.json` files directly from the `agent/app/` directly, without putting them into a sub-directory:

### Proposed API / Implementation

Add an `ADK_EVALS_DIR` env var that sets a subdirectory name to use for eval files within the agent directory. When unset, behaviour is identical to today. This gives users explicit control without changing any defaults.

```python
_EVAL_SET_FILE_EXTENSION = ".evalset.json"
_EVALS_SUBDIR_ENV_VAR = "ADK_EVALS_DIR" # e.g. "evals" or "eval"

class LocalEvalSetsManager(EvalSetsManager):

def __init__(self, agents_dir: str):
self._agents_dir = agents_dir
self._evals_subdir = self._load_evals_subdir()

def _load_evals_subdir(self) -> str:
evals_subdir = os.environ.get(_EVALS_SUBDIR_ENV_VAR, "")
if evals_subdir:
if os.path.isabs(evals_subdir):
raise ValueError(
f"{_EVALS_SUBDIR_ENV_VAR} must be a relative path,"
f" got: {evals_subdir!r}"
)
if ".." in evals_subdir.split(os.sep):
raise ValueError(
f"{_EVALS_SUBDIR_ENV_VAR} must not contain '..',"
f" got: {evals_subdir!r}"
)
return evals_subdir

def _get_eval_base_dir(self, app_name: str) -> str:
base = os.path.join(self._agents_dir, app_name, self._evals_subdir)
resolved = os.path.realpath(base)
agents_dir = os.path.realpath(self._agents_dir)
if not resolved.startswith(agents_dir + os.sep):
raise ValueError(f"Invalid app_name: {app_name!r}")
return resolved

def _get_eval_set_file_path(self, app_name: str, eval_set_id: str) -> str:
return os.path.join(
self._get_eval_base_dir(app_name),
eval_set_id + _EVAL_SET_FILE_EXTENSION,
)

def list_eval_sets(self, app_name: str) -> list[str]:
eval_base_dir = self._get_eval_base_dir(app_name)
try:
return sorted(
f.removesuffix(_EVAL_SET_FILE_EXTENSION)
for f in os.listdir(eval_base_dir)
if f.endswith(_EVAL_SET_FILE_EXTENSION)
)
except FileNotFoundError as e:
raise NotFoundError(
f"Eval directory for app `{app_name}` not found."
) from e
```

Usage:
```bash
ADK_EVALS_DIR=evals adk web path/to/agents_dir
```

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.