Evals fail with VertexAiSessionService because generated session IDs violate Agent Engine constraints
- Dominant language
- Python
- Stars
- 21.5k
- Forks
- 4k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 37
Description
## 🔴 Required Information
**Describe the Bug:**
Running an evaluation with `VertexAiSessionService` fails because ADK generates evaluation session IDs using the following format:
```text
___eval___session___
```
The underscores in this ID violate the Vertex AI Agent Engine Sessions restrictions for custom session IDs. Agent Engine only accepts lowercase letters, digits, and hyphens (`[a-z0-9-]`), with a letter or digit as the first and last character.
The eval session ID is generated in `google/adk/evaluation/local_eval_service.py`:
https://github.com/google/adk-python/blob/0b55dcf9d32e22d4c8b303c3da1c275c135682bf/src/google/adk/evaluation/local_eval_service.py#L64-L68
It is then passed to the configured session service by `_get_or_create_eval_session()` in `google/adk/evaluation/evaluation_generator.py`:
https://github.com/google/adk-python/blob/0b55dcf9d32e22d4c8b303c3da1c275c135682bf/src/google/adk/evaluation/evaluation_generator.py#L118-L123
When the configured session service is `VertexAiSessionService`, Agent Engine rejects the generated ID and the evaluation cannot start.
There is also a validation mismatch in `VertexAiSessionService`. Its current local pattern accepts underscores and uppercase letters:
ref: https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/sessions/manage-with-api?hl=en#create_a_session
```python
_SESSION_ID_PATTERN = re.compile(r'^[A-Za-z0-9_-]+$')
```
Therefore, the invalid eval session ID passes ADK-side validation and is rejected only by the remote Agent Engine API.
**Steps to Reproduce:**
1. Install ADK with the required extras:
```bash
pip install "google-adk[gcp,eval]==2.6.3"
```
2. Configure an Agent Engine-backed session service:
```bash
adk web \
--session_service_uri="agentengine://projects//locations//reasoningEngines/" \
.
```
3. Open the ADK Web UI and create or select an eval set.
4. Run an evaluation.
5. Observe that creation of the generated `___eval___session___` session fails.
The problem can also be isolated through the session creation endpoint:
| Request | Result |
|---|---|
| No custom session ID | `200 OK` |
| `eval-` | `200 OK` |
| `___eval___session___` | `500 Internal Server Error` |
**Expected Behavior:**
Evaluations should be able to **create** their temporary sessions when `VertexAiSessionService` is configured.
**Observed Behavior:**
Agent Engine rejects the generated session ID:
```text
400 INVALID_ARGUMENT
session_id can only contain lowercase letters, digits and hyphens.
The first and last characters must be a letter or number.
```
The exception is subsequently exposed by the ADK Web/API server as an HTTP 500, and the evaluation fails before inference begins.
**Environment Details:**
- ADK Library Version (pip show google-adk):
- `google-adk==1.38.0`
- `google-adk==2.6.3`
- Desktop OS: macOS 26.4.1
- Python Version: Python 3.13.12
**Model Information:**
- Are you using LiteLLM: No
- Which model is being used: N/A — the failure occurs before model inference
---
## 🟡 Optional Information
**Regression:**
This is related to #2731, which reported that `VertexAiSessionService` did not support user-provided session IDs.
Current ADK releases now pass custom IDs to Agent Engine. However, evals still fail because the ADK-generated `___eval___session___` prefix does not satisfy the Agent Engine ID format.
Issue #2731 was closed as stale rather than by a merged fix, so this appears to be the remaining/current form of the compatibility problem.
**Logs:**
```text
400 INVALID_ARGUMENT. {
'error': {
'code': 400,
'message': (
'List of found errors: '
'1.Field: session_id; Message: session_id can only contain '
'lowercase letters, digits and hyphens. The first and last '
'characters must be a letter or number.'
),
'status': 'INVALID_ARGUMENT'
}
}
```
**Screenshots / Video:**
N/A
**Additional Context:**
I developed `adk eval ` version, and I found this issue.
https://github.com/ftnext/pytest-adk/releases/tag/0.0.7
**Minimal Reproduction Code:**
```python
import asyncio
from google.adk.sessions import VertexAiSessionService
async def main() -> None:
service = VertexAiSessionService(
project="",
location="",
agent_engine_id="",
)
await service.create_session(
app_name="",
user_id="test-user",
session_id="___eval___session___550e8400-e29b-41d4-a716-446655440000",
)
asyncio.run(main())
```
**How often has this issue occurred?:**
- Always (100%)
Contributor guide
Assessment
This issue has not been assessed yet.