awslabs / awslabs/agent-evaluation

Formalize test plan schema

Open
#21 3 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue
Dominant language
Python
Stars
372
Forks
51
PR merge metrics
No merged PRs in 30d

Description

Currently test plan is defined as yaml file, for example for the target field

```
target:
type: bedrock-agent
bedrock_agent_id: <>
bedrock_agent_alias_id: <>
```

If converting the current yaml to json schema it would be like:

```
"target": {
"type": "object",
"properties": {
"type": {"type": "string"},
"bedrock_agent_id": {"type": "string"},
"bedrock_agent_alias_id": {"type": "string"}
},
"required": ["type", "bedrock_agent_id", "bedrock_agent_alias_id"]
}
```
which doesn't work with other target type making it hard to validate the test plan without actually running the test. Proposing to have a json schema defined for the test plan file like:

```
"target": {
"type": "object",
"properties": {
"type": {
"type": "string",
"enum": ["bedrock-agent", "q-business"]
},
"attributes": {
"oneOf": [
{
"properties": {
"bedrock_agent_id": {"type": "string"},
"bedrock_agent_alias_id": {"type": "string"}
},
"required": ["bedrock_agent_id", "bedrock_agent_alias_id"],
"if": {"properties": {"type": {"const": "bedrock-agent"}}}
},
{
"properties": {
"q_business_application_id": {"type": "string"},
"q_business_user_id": {"type": "string"}
},
"required": ["q_business_application_id", "q_business_user_id"],
"if": {"properties": {"type": {"const": "q-business"}}}
}
]
}
},
"required": ["type", "attributes"]
}
```

so that:
1. yaml and json test plan can be interchangeable
2. test plan file can be validated on the format

Acceptance criteria:
1. json format for the test plan file in-place
2. when triggering the runner, validate the test plan format and throw clear errors if malformatted
3. add unit test

Contributor guide

Open the contributing guide

Research direction

Start by locating the test plan parser and the runner entry point, then inspect how YAML plans are currently loaded and where validation errors are reported. Add the JSON representation and format validation for runner-triggered plans, and add the requested unit test covering valid and malformed plans.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.