google / google/adk-python

Align `adk web` eval logic with `adk eval` CLI & `pytest`

Đang mở
#4,867 1 bình luận 2 reaction 2 người được giao Được @Jacksunwei nhận Xem trên GitHub
eval needs review
Ngôn ngữ chính
Python
Star
21.5k
Fork
4k
Merge trung bình
1 ngày 14 giờ
Pull request đã merge (30 ngày)
37

Mô tả

### Is your feature request related to a specific problem?
`adk web` eval discovery is inconsistent with `adk eval` and pytest workflows

There are three ways to run evals in ADK, from what I can tell, and only one of them constrains where eval files must live:

- **`adk eval`** accepts eval set file paths as direct positional arguments — files can be anywhere on disk:
```bash
adk eval my_agent path/to/evals/my_eval.evalset.json
```
- **pytest** (via `AgentEvaluator.evaluate()`) accepts a file path as a parameter — files can be anywhere on disk:
```python
await AgentEvaluator.evaluate("my_agent", "path/to/evals/my_eval.evalset.json")
```
- **`adk web`** uses `LocalEvalSetsManager`, which hardcodes discovery to a flat `os.listdir` on `//` with no way to override it. The only alternative is `--eval_storage_uri=gs://...` for GCS — there is no local path equivalent. Files must sit directly at `//*.evalset.json`.

This means a project following the convention from Google's own [adk-samples](https://github.com/google/adk-samples/tree/main/python/agents) — where eval files live in an `/eval/` subdirectory and are run via pytest or `adk eval` — gets no visibility into those evals from the `adk web` UI. Eval sets created through the `adk web` UI are also written to the agent root via `_get_eval_set_file_path`, diverging from where the project's other eval files are stored.

### Describe the Solution You'd Like
Allow for `adk web` to be configured (through an environment variable or otherwise) with a path where `*.evalset.json` files should be created and read from.

### Impact on your work
Cannot provide a coherent developer experience where JSON files are created through the web UI and then run in CI/CD without polluting the agent app directory with `evalset.json` files.

### Willingness to contribute
Are you interested in implementing this feature yourself or submitting a PR?
Maybe

### Describe Alternatives You've Considered
Using `evalset.json` files directly from the `agent/app/` directly, without putting them into a sub-directory:

### Proposed API / Implementation

Add an `ADK_EVALS_DIR` env var that sets a subdirectory name to use for eval files within the agent directory. When unset, behaviour is identical to today. This gives users explicit control without changing any defaults.

```python
_EVAL_SET_FILE_EXTENSION = ".evalset.json"
_EVALS_SUBDIR_ENV_VAR = "ADK_EVALS_DIR" # e.g. "evals" or "eval"

class LocalEvalSetsManager(EvalSetsManager):

def __init__(self, agents_dir: str):
self._agents_dir = agents_dir
self._evals_subdir = self._load_evals_subdir()

def _load_evals_subdir(self) -> str:
evals_subdir = os.environ.get(_EVALS_SUBDIR_ENV_VAR, "")
if evals_subdir:
if os.path.isabs(evals_subdir):
raise ValueError(
f"{_EVALS_SUBDIR_ENV_VAR} must be a relative path,"
f" got: {evals_subdir!r}"
)
if ".." in evals_subdir.split(os.sep):
raise ValueError(
f"{_EVALS_SUBDIR_ENV_VAR} must not contain '..',"
f" got: {evals_subdir!r}"
)
return evals_subdir

def _get_eval_base_dir(self, app_name: str) -> str:
base = os.path.join(self._agents_dir, app_name, self._evals_subdir)
resolved = os.path.realpath(base)
agents_dir = os.path.realpath(self._agents_dir)
if not resolved.startswith(agents_dir + os.sep):
raise ValueError(f"Invalid app_name: {app_name!r}")
return resolved

def _get_eval_set_file_path(self, app_name: str, eval_set_id: str) -> str:
return os.path.join(
self._get_eval_base_dir(app_name),
eval_set_id + _EVAL_SET_FILE_EXTENSION,
)

def list_eval_sets(self, app_name: str) -> list[str]:
eval_base_dir = self._get_eval_base_dir(app_name)
try:
return sorted(
f.removesuffix(_EVAL_SET_FILE_EXTENSION)
for f in os.listdir(eval_base_dir)
if f.endswith(_EVAL_SET_FILE_EXTENSION)
)
except FileNotFoundError as e:
raise NotFoundError(
f"Eval directory for app `{app_name}` not found."
) from e
```

Usage:
```bash
ADK_EVALS_DIR=evals adk web path/to/agents_dir
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.