Support pre-populated session events in `SessionInput` for eval cases
- Dominant language
- Python
- Stars
- 21.5k
- Forks
- 4k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 37
Description
## 🔴 Required Information
### Is your feature request related to a specific problem?
The eval framework's documentation describes test files as "unit testing" individual agent behaviors, but there is no way to test a conversation turn that depends on prior context.
For example, consider an agent that can send SMS. A user says "Can you text me that information?". This only makes sense after a prior exchange where the agent looked up some information. Today there is no way to express this in an eval test file: `SessionInput` only accepts `app_name`, `user_id`, and `state`.
If I put the context turns in `conversation`, they are all scored alongside the turn I actually want to test, adding noise and fragility (non-deterministic model responses on context turns drag down `response_match_score`). If I omit context turns, the agent has no conversation history and "text me that information" has no referent.
### Describe the Solution You'd Like
Add an optional `events` field to `SessionInput` that accepts a list of `Event` objects. After `create_session`, the eval framework should append these events to the session before starting inference. This seeds the session with conversation history so that eval turns run with realistic prior context.
```python
class SessionInput(EvalBaseModel):
app_name: str
user_id: str
state: SessionState = Field(default_factory=dict)
events: Optional[list[Event]] = None # <-- new field
```
In `EvaluationGenerator._generate_inferences_from_root_agent`, after the `create_session` call, append the seed events:
```python
session = await session_service.create_session(
app_name=app_name,
user_id=user_id,
state=initial_session.state if initial_session else {},
session_id=session_id,
)
# Seed session with prior conversation history
if initial_session and initial_session.events:
for event in initial_session.events:
await session_service.append_event(session=session, event=event)
```
The `conversation` array continues to define only the turns that are replayed and scored.
### Impact on your work
Many behaviors we need to eval (sending SMS, confirming actions, following up on earlier answers, etc.) only make sense mid-conversation. Without session history seeding, we cannot write targeted eval cases for these behaviors and must either skip eval coverage or accept noisy multi-turn evals where context turns are scored unnecessarily.
### Willingness to contribute
Yes.
---
## 🟡 Recommended Information
### Describe Alternatives You've Considered
1. **Put all turns (context + test) in `conversation`**: Every entry is both replayed and scored. Context turns add noise — `response_match_score` (ROUGE-1) is fragile on non-deterministic model output, and `tool_trajectory_avg_score` averages across all turns, so a context-turn mismatch drags down the overall score. This also means authoring and maintaining expected responses for turns you don't actually care about testing.
2. **Encode context in `session_input.state`**: The agent's LLM does not see session state as conversation history — it sees `session.events`. Putting a summary in state doesn't give the model the actual prior turns in its context window, so context-dependent references like "text me *that*" don't resolve.
### Proposed API / Implementation
**JSON test file** with seed events in `session_input`:
```json
{
"eval_set_id": "agent_send_sms",
"eval_cases": [
{
"eval_id": "send_sms_after_lookup",
"conversation": [
{
"invocation_id": "test-turn-1",
"user_content": {
"parts": [{"text": "Can you text me that information?"}],
"role": "user"
},
"final_response": {
"parts": [{"text": "I've sent you a text with the details."}],
"role": "model"
},
"intermediate_data": {
"tool_uses": [
{"name": "send_sms", "args": {"body": "..."}}
],
"intermediate_responses": []
}
}
],
"session_input": {
"app_name": "my_agent",
"user_id": "test_user",
"state": {},
"events": [
{
"content": {"parts": [{"text": "How do I pay my water bill?"}], "role": "user"},
"author": "user",
"invocation_id": "seed-0"
},
{
"content": {"parts": [{"function_call": {"name": "document_search", "args": {"query": "pay water bill"}}}], "role": "model"},
"author": "root_agent",
"invocation_id": "seed-0"
},
{
"content": {"parts": [{"function_response": {"name": "document_search", "response": {"result": "Call ABC at (555) 555-5555..."}}}], "role": "user"},
"author": "root_agent",
"invocation_id": "seed-0"
},
{
"content": {"parts": [{"text": "You can pay your water bill by calling ABC at (555) 555-5555."}], "role": "model"},
"author": "root_agent",
"invocation_id": "seed-0"
}
]
}
}
]
}
```
**Framework changes** (minimal):
1. Add `events: Optional[list[Event]] = None` to `SessionInput` in `eval_case.py`.
2. In `EvaluationGenerator._generate_inferences_from_root_agent`, append seed events after `create_session`.
3. No changes needed to scoring — `conversation` still defines the scored turns.
### Additional Context
- The eval docs describe test files as "a form of unit testing" ([source](https://google.github.io/adk-docs/evaluate/#first-approach-using-a-test-file)), but the current design only supports testing turns from a cold start.
- The `EvalCase` model already has `final_session_state` for asserting post-eval state, so there is precedent for session-level fields beyond just conversation turns.
Contributor guide
Assessment
This issue has not been assessed yet.