google / google/adk-python

Support pre-populated session events in `SessionInput` for eval cases

Đang mở
#4,896 1 bình luận 0 reaction 1 người được giao Được @GWeale nhận Xem trên GitHub
eval needs review
Ngôn ngữ chính
Python
Star
21.5k
Fork
4k
Merge trung bình
1 ngày 14 giờ
Pull request đã merge (30 ngày)
37

Mô tả

## 🔴 Required Information

### Is your feature request related to a specific problem?

The eval framework's documentation describes test files as "unit testing" individual agent behaviors, but there is no way to test a conversation turn that depends on prior context.

For example, consider an agent that can send SMS. A user says "Can you text me that information?". This only makes sense after a prior exchange where the agent looked up some information. Today there is no way to express this in an eval test file: `SessionInput` only accepts `app_name`, `user_id`, and `state`.

If I put the context turns in `conversation`, they are all scored alongside the turn I actually want to test, adding noise and fragility (non-deterministic model responses on context turns drag down `response_match_score`). If I omit context turns, the agent has no conversation history and "text me that information" has no referent.

### Describe the Solution You'd Like

Add an optional `events` field to `SessionInput` that accepts a list of `Event` objects. After `create_session`, the eval framework should append these events to the session before starting inference. This seeds the session with conversation history so that eval turns run with realistic prior context.

```python
class SessionInput(EvalBaseModel):
app_name: str
user_id: str
state: SessionState = Field(default_factory=dict)
events: Optional[list[Event]] = None # <-- new field
```

In `EvaluationGenerator._generate_inferences_from_root_agent`, after the `create_session` call, append the seed events:

```python
session = await session_service.create_session(
app_name=app_name,
user_id=user_id,
state=initial_session.state if initial_session else {},
session_id=session_id,
)

# Seed session with prior conversation history
if initial_session and initial_session.events:
for event in initial_session.events:
await session_service.append_event(session=session, event=event)
```

The `conversation` array continues to define only the turns that are replayed and scored.

### Impact on your work

Many behaviors we need to eval (sending SMS, confirming actions, following up on earlier answers, etc.) only make sense mid-conversation. Without session history seeding, we cannot write targeted eval cases for these behaviors and must either skip eval coverage or accept noisy multi-turn evals where context turns are scored unnecessarily.

### Willingness to contribute

Yes.

---

## 🟡 Recommended Information

### Describe Alternatives You've Considered

1. **Put all turns (context + test) in `conversation`**: Every entry is both replayed and scored. Context turns add noise — `response_match_score` (ROUGE-1) is fragile on non-deterministic model output, and `tool_trajectory_avg_score` averages across all turns, so a context-turn mismatch drags down the overall score. This also means authoring and maintaining expected responses for turns you don't actually care about testing.

2. **Encode context in `session_input.state`**: The agent's LLM does not see session state as conversation history — it sees `session.events`. Putting a summary in state doesn't give the model the actual prior turns in its context window, so context-dependent references like "text me *that*" don't resolve.

### Proposed API / Implementation

**JSON test file** with seed events in `session_input`:

```json
{
"eval_set_id": "agent_send_sms",
"eval_cases": [
{
"eval_id": "send_sms_after_lookup",
"conversation": [
{
"invocation_id": "test-turn-1",
"user_content": {
"parts": [{"text": "Can you text me that information?"}],
"role": "user"
},
"final_response": {
"parts": [{"text": "I've sent you a text with the details."}],
"role": "model"
},
"intermediate_data": {
"tool_uses": [
{"name": "send_sms", "args": {"body": "..."}}
],
"intermediate_responses": []
}
}
],
"session_input": {
"app_name": "my_agent",
"user_id": "test_user",
"state": {},
"events": [
{
"content": {"parts": [{"text": "How do I pay my water bill?"}], "role": "user"},
"author": "user",
"invocation_id": "seed-0"
},
{
"content": {"parts": [{"function_call": {"name": "document_search", "args": {"query": "pay water bill"}}}], "role": "model"},
"author": "root_agent",
"invocation_id": "seed-0"
},
{
"content": {"parts": [{"function_response": {"name": "document_search", "response": {"result": "Call ABC at (555) 555-5555..."}}}], "role": "user"},
"author": "root_agent",
"invocation_id": "seed-0"
},
{
"content": {"parts": [{"text": "You can pay your water bill by calling ABC at (555) 555-5555."}], "role": "model"},
"author": "root_agent",
"invocation_id": "seed-0"
}
]
}
}
]
}
```

**Framework changes** (minimal):

1. Add `events: Optional[list[Event]] = None` to `SessionInput` in `eval_case.py`.
2. In `EvaluationGenerator._generate_inferences_from_root_agent`, append seed events after `create_session`.
3. No changes needed to scoring — `conversation` still defines the scored turns.

### Additional Context

- The eval docs describe test files as "a form of unit testing" ([source](https://google.github.io/adk-docs/evaluate/#first-approach-using-a-test-file)), but the current design only supports testing turns from a cold start.
- The `EvalCase` model already has `final_session_state` for asserting post-eval state, so there is precedent for session-level fields beyond just conversation turns.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.