google / google/adk-python

Feature Request: Support Custom Evalset Storage Directory via Environment Variable

Abierto
#3,887 3 comentarios 4 reacciones 2 asignados Reclamado por @klateefa Ver en GitHub
eval needs review planned
Lenguaje dominante
Python
Estrellas
21.5k
Forks
4k
Merge medio
1 d 14 h
PR fusionados (30 d)
37

Descripción

**Is your feature request related to a problem? Please describe.**

When deploying ADK applications in Kubernetes or containerized environments, the `agents_dir` is often read-only (e.g., baked into the container image), but evaluation storage requires write access. Currently, `LocalEvalSetsManager` and `LocalEvalSetResultsManager` hardcode the use of `agents_dir` for storing evalsets and results, which causes permission errors in production environments where:

1. **K8s deployments** mount application code as read-only ConfigMaps or from container images
2. **Security policies** restrict write access to application directories
3. **Ephemeral storage** (like `/tmp`) is the only writable location available

This forces users to choose between:
- Complex workarounds (monkey-patching ADK internals)
- Compromising security by making the entire agents directory writable
- Not using the eval feature at all in production

**Example error scenario:**
```python
# In K8s pod where /app is read-only
app = get_fast_api_app(
agents_dir="/app/agents", # Read-only
eval_storage_uri=None, # Defaults to agents_dir
)

# Results in:
# PermissionError: [Errno 13] Permission denied: '/app/agents/my-app/my-evalset.evalset.json'
```

**Describe the solution you'd like**

Add support for a dedicated eval storage directory that's separate from `agents_dir`, configurable via:

### Option 1: Environment Variable (Preferred)
```bash
# Set custom eval storage location
export ADK_EVAL_STORAGE_DIR="/tmp/adk_evals"

# Or use existing eval_storage_uri pattern
export ADK_EVAL_STORAGE_URI="file: ///tmp/adk_evals"
```

```python
# Automatically uses ADK_EVAL_STORAGE_DIR if set
app = get_fast_api_app(
agents_dir="/app/agents", # Read-only, contains agent definitions
# eval storage automatically uses /tmp/adk_evals from env var
)
```

### Option 2: Explicit Parameter
```python
app = get_fast_api_app(
agents_dir="/app/agents", # Read-only
eval_storage_dir="/tmp/adk_evals", # Writable
)
```

### Option 3: Extend eval_storage_uri to Support file:// URIs
Currently `eval_storage_uri` only supports `gs://` (GCS). Extend it to support local file URIs:

```python
app = get_fast_api_app(
agents_dir="/app/agents",
eval_storage_uri="file:///tmp/adk_evals", # Currently not supported
)
```

**Implementation in `fast_api.py`:**
```python
def get_fast_api_app(
*,
agents_dir: str,
eval_storage_uri: Optional[str] = None,
eval_storage_dir: Optional[str] = None, # New parameter
...
):
# Priority: explicit param > env var > eval_storage_uri > default to agents_dir
eval_dir = (
eval_storage_dir
or os.getenv("ADK_EVAL_STORAGE_DIR")
or agents_dir
)

if eval_storage_uri:
if eval_storage_uri.startswith('gs://'):
# Existing GCS logic
...
elif eval_storage_uri.startswith('file://'):
# New: Support local file URIs
eval_dir = eval_storage_uri. replace('file://', '')

eval_sets_manager = LocalEvalSetsManager(agents_dir=eval_dir)
eval_set_results_manager = LocalEvalSetResultsManager(agents_dir=eval_dir)
```

**Describe alternatives you've considered**

1. **Monkey-patching (Current Workaround)**
- Patching `LocalEvalSetsManager.__init__` and `LocalEvalSetResultsManager.__init__` to override `agents_dir`
- **Problems**:
- Fragile (breaks with ADK updates)
- Hard to maintain
- Not officially supported
- Timing-sensitive (must patch before ADK imports)

2. **Using GCS with eval_storage_uri**
- Works but requires GCS setup, credentials, and network access
- **Problems**:
- Overkill for local/development use
- Adds infrastructure complexity
- Requires GCS bucket, IAM permissions
- Not suitable for air-gapped environments

3. **Making agents_dir writable**
- Violates security best practices
- **Problems**:
- Allows modification of agent code at runtime
- Increases attack surface
- Goes against K8s immutable infrastructure principles

4. **Separate eval-only deployment**
- Deploy a separate ADK instance just for evals with writable agents_dir
- **Problems**:
- Resource waste
- Deployment complexity
- Data synchronization challenges

**Additional context**

### Use Case: Production K8s Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: adk-agent-service
spec:
template:
spec:
containers:
- name: app
image: my-adk-app: latest
env:
- name: ADK_EVAL_STORAGE_DIR # Proposed solution
value: "/tmp/adk_evals"
volumeMounts:
- name: agent-code
mountPath: /app/agents
readOnly: true # Security requirement
- name: eval-storage
mountPath: /tmp/adk_evals
volumes:
- name: agent-code
configMap:
name: agent-definitions
- name: eval-storage
emptyDir: {} # Writable ephemeral storage
```

### Current File Structure Issues
```
/app/agents/ # Read-only in production
├── my-app/
│ ├── agent. py # Agent code
│ ├── root_agent.yaml # Agent config
│ └── my-evalset.evalset.json # ❌ FAILS: Can't write here in K8s
```

### Desired File Structure
```
/app/agents/ # Read-only (agent definitions)
├── my-app/
│ ├── agent.py
│ └── root_agent.yaml

/tmp/adk_evals/ # Writable (eval storage)
├── my-app/
│ └── my-evalset.evalset.json # ✅ SUCCESS: Writable location
```

### Related Code Locations (ADK 1.18+)
- `src/google/adk/cli/fast_api.py:105-114` - Where eval managers are initialized
- `src/google/adk/cli/utils/evals.py:45-66` - `create_gcs_eval_managers_from_uri()` only supports `gs://`
- `src/google/adk/evaluation/local_eval_sets_manager.py` - Uses `agents_dir` directly
- `src/google/adk/evaluation/local_eval_set_results_manager.py` - Uses `agents_dir` directly

### Compatibility Considerations
- **Backward compatibility**: Default to `agents_dir` if no custom location specified
- **Path resolution**: Support both absolute and relative paths
- **Directory creation**: Auto-create eval storage directory with proper permissions
- **Migration**: Existing evalsets in `agents_dir` should still be readable

### Expected Behavior
```python
# Development (agents_dir is writable)
app = get_fast_api_app(agents_dir="./agents")
# Evals stored in: ./agents/{app_name}/

# Production (custom eval storage)
app = get_fast_api_app(
agents_dir="/app/agents", # Read-only
eval_storage_dir="/tmp/adk_evals" # Writable
)
# Agent definitions read from: /app/agents/
# Evals stored in: /tmp/adk_evals/{app_name}/
```

This feature would make ADK significantly easier to deploy in production environments following security and immutability best practices, while maintaining backward compatibility for existing deployments.

---

**ADK Version:** 1.18 (also affects 1.19, 1.20)

**Environment:** Kubernetes, Docker, containerized deployments

**Related Issues:** N/A (first report of this specific use case)

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.