google / google/adk-python

Feature Request: Support Custom Evalset Storage Directory via Environment Variable

オープン
#3,887 コメント 3 件 リアクション 4 件 担当者 2 名 @klateefa が担当を希望しています GitHub で見る
eval needs review planned
主要言語
Python
スター
21.5k
フォーク
4k
平均マージ
1日 14時間
マージ済み PR(30日)
37

説明

**Is your feature request related to a problem? Please describe.**

When deploying ADK applications in Kubernetes or containerized environments, the `agents_dir` is often read-only (e.g., baked into the container image), but evaluation storage requires write access. Currently, `LocalEvalSetsManager` and `LocalEvalSetResultsManager` hardcode the use of `agents_dir` for storing evalsets and results, which causes permission errors in production environments where:

1. **K8s deployments** mount application code as read-only ConfigMaps or from container images
2. **Security policies** restrict write access to application directories
3. **Ephemeral storage** (like `/tmp`) is the only writable location available

This forces users to choose between:
- Complex workarounds (monkey-patching ADK internals)
- Compromising security by making the entire agents directory writable
- Not using the eval feature at all in production

**Example error scenario:**
```python
# In K8s pod where /app is read-only
app = get_fast_api_app(
agents_dir="/app/agents", # Read-only
eval_storage_uri=None, # Defaults to agents_dir
)

# Results in:
# PermissionError: [Errno 13] Permission denied: '/app/agents/my-app/my-evalset.evalset.json'
```

**Describe the solution you'd like**

Add support for a dedicated eval storage directory that's separate from `agents_dir`, configurable via:

### Option 1: Environment Variable (Preferred)
```bash
# Set custom eval storage location
export ADK_EVAL_STORAGE_DIR="/tmp/adk_evals"

# Or use existing eval_storage_uri pattern
export ADK_EVAL_STORAGE_URI="file: ///tmp/adk_evals"
```

```python
# Automatically uses ADK_EVAL_STORAGE_DIR if set
app = get_fast_api_app(
agents_dir="/app/agents", # Read-only, contains agent definitions
# eval storage automatically uses /tmp/adk_evals from env var
)
```

### Option 2: Explicit Parameter
```python
app = get_fast_api_app(
agents_dir="/app/agents", # Read-only
eval_storage_dir="/tmp/adk_evals", # Writable
)
```

### Option 3: Extend eval_storage_uri to Support file:// URIs
Currently `eval_storage_uri` only supports `gs://` (GCS). Extend it to support local file URIs:

```python
app = get_fast_api_app(
agents_dir="/app/agents",
eval_storage_uri="file:///tmp/adk_evals", # Currently not supported
)
```

**Implementation in `fast_api.py`:**
```python
def get_fast_api_app(
*,
agents_dir: str,
eval_storage_uri: Optional[str] = None,
eval_storage_dir: Optional[str] = None, # New parameter
...
):
# Priority: explicit param > env var > eval_storage_uri > default to agents_dir
eval_dir = (
eval_storage_dir
or os.getenv("ADK_EVAL_STORAGE_DIR")
or agents_dir
)

if eval_storage_uri:
if eval_storage_uri.startswith('gs://'):
# Existing GCS logic
...
elif eval_storage_uri.startswith('file://'):
# New: Support local file URIs
eval_dir = eval_storage_uri. replace('file://', '')

eval_sets_manager = LocalEvalSetsManager(agents_dir=eval_dir)
eval_set_results_manager = LocalEvalSetResultsManager(agents_dir=eval_dir)
```

**Describe alternatives you've considered**

1. **Monkey-patching (Current Workaround)**
- Patching `LocalEvalSetsManager.__init__` and `LocalEvalSetResultsManager.__init__` to override `agents_dir`
- **Problems**:
- Fragile (breaks with ADK updates)
- Hard to maintain
- Not officially supported
- Timing-sensitive (must patch before ADK imports)

2. **Using GCS with eval_storage_uri**
- Works but requires GCS setup, credentials, and network access
- **Problems**:
- Overkill for local/development use
- Adds infrastructure complexity
- Requires GCS bucket, IAM permissions
- Not suitable for air-gapped environments

3. **Making agents_dir writable**
- Violates security best practices
- **Problems**:
- Allows modification of agent code at runtime
- Increases attack surface
- Goes against K8s immutable infrastructure principles

4. **Separate eval-only deployment**
- Deploy a separate ADK instance just for evals with writable agents_dir
- **Problems**:
- Resource waste
- Deployment complexity
- Data synchronization challenges

**Additional context**

### Use Case: Production K8s Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: adk-agent-service
spec:
template:
spec:
containers:
- name: app
image: my-adk-app: latest
env:
- name: ADK_EVAL_STORAGE_DIR # Proposed solution
value: "/tmp/adk_evals"
volumeMounts:
- name: agent-code
mountPath: /app/agents
readOnly: true # Security requirement
- name: eval-storage
mountPath: /tmp/adk_evals
volumes:
- name: agent-code
configMap:
name: agent-definitions
- name: eval-storage
emptyDir: {} # Writable ephemeral storage
```

### Current File Structure Issues
```
/app/agents/ # Read-only in production
├── my-app/
│ ├── agent. py # Agent code
│ ├── root_agent.yaml # Agent config
│ └── my-evalset.evalset.json # ❌ FAILS: Can't write here in K8s
```

### Desired File Structure
```
/app/agents/ # Read-only (agent definitions)
├── my-app/
│ ├── agent.py
│ └── root_agent.yaml

/tmp/adk_evals/ # Writable (eval storage)
├── my-app/
│ └── my-evalset.evalset.json # ✅ SUCCESS: Writable location
```

### Related Code Locations (ADK 1.18+)
- `src/google/adk/cli/fast_api.py:105-114` - Where eval managers are initialized
- `src/google/adk/cli/utils/evals.py:45-66` - `create_gcs_eval_managers_from_uri()` only supports `gs://`
- `src/google/adk/evaluation/local_eval_sets_manager.py` - Uses `agents_dir` directly
- `src/google/adk/evaluation/local_eval_set_results_manager.py` - Uses `agents_dir` directly

### Compatibility Considerations
- **Backward compatibility**: Default to `agents_dir` if no custom location specified
- **Path resolution**: Support both absolute and relative paths
- **Directory creation**: Auto-create eval storage directory with proper permissions
- **Migration**: Existing evalsets in `agents_dir` should still be readable

### Expected Behavior
```python
# Development (agents_dir is writable)
app = get_fast_api_app(agents_dir="./agents")
# Evals stored in: ./agents/{app_name}/

# Production (custom eval storage)
app = get_fast_api_app(
agents_dir="/app/agents", # Read-only
eval_storage_dir="/tmp/adk_evals" # Writable
)
# Agent definitions read from: /app/agents/
# Evals stored in: /tmp/adk_evals/{app_name}/
```

This feature would make ADK significantly easier to deploy in production environments following security and immutability best practices, while maintaining backward compatibility for existing deployments.

---

**ADK Version:** 1.18 (also affects 1.19, 1.20)

**Environment:** Kubernetes, Docker, containerized deployments

**Related Issues:** N/A (first report of this specific use case)

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。