aws-samples / aws-samples/sample-agent-greenhouse

Add fail-fast harness validation at agent construction time

Open
#6 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
21
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Problem

`DomainHarness.__post_init__` only validates `if not self.name`. There is no validation that:

- Referenced skills exist in the skill registry
- Hook dotted-paths (`HookConfig`) are importable
- Tool allowlists in `PolicyConfig` reference registered tools
- Memory namespace templates contain required placeholders like `{actorId}`
- Evaluation rule scorer references are valid

Misconfigurations in the harness YAML currently surface as **runtime errors deep in the hook chain** — sometimes minutes into an agent session. This results in cryptic `AttributeError` or `ImportError` messages that are difficult to debug.

## Proposed Solution

Add a `validate()` method on `DomainHarness` and call it in `FoundationAgent.__init__`:

```python
@dataclass(frozen=True)
class DomainHarness:
# ... existing fields ...

def validate(self, skill_registry=None, tool_registry=None) -> list[str]:
"""Validate harness configuration. Returns list of error messages."""
errors = []

# 1. Validate skill references
if skill_registry:
for skill_ref in self.skills:
if skill_ref.name not in skill_registry:
errors.append(f"Skill '{skill_ref.name}' not found in registry")

# 2. Validate hook imports
for hook_config in self.hooks:
try:
importlib.import_module(hook_config.module_path)
except ImportError:
errors.append(f"Hook '{hook_config.module_path}' cannot be imported")

# 3. Validate memory namespace templates
if self.memory and self.memory.namespace_template:
if "{actorId}" not in self.memory.namespace_template:
errors.append("Memory namespace_template must contain {actorId}")

# 4. Validate policy tool references
if tool_registry and self.policies:
for tool in self.policies.tool_allowlist:
if tool not in tool_registry:
errors.append(f"Policy references unknown tool '{tool}'")

return errors
```

In `FoundationAgent.__init__`:
```python
errors = harness.validate(skill_registry=self._skill_registry)
if errors:
raise HarnessValidationError(
f"Invalid harness configuration:\n" +
"\n".join(f" - {e}" for e in errors)
)
```

## Impact

This is the **single highest-impact improvement for developer experience**. It turns:
```
AttributeError: 'NoneType' object has no attribute 'execute'
at hooks/tool_policy_hook.py:87 (3 minutes into session)
```
Into:
```
HarnessValidationError: Invalid harness configuration:
- Skill 'nonexistent_skill' not found in registry
- Hook 'platform_agent.hooks.missing' cannot be imported
```

## Acceptance Criteria

- [ ] `DomainHarness.validate()` method checks skills, hooks, tools, memory, and eval rules
- [ ] `FoundationAgent.__init__` calls `validate()` and raises `HarnessValidationError` on failure
- [ ] Error messages are actionable (include the invalid value and what was expected)
- [ ] `from_yaml()` optionally validates during deserialization
- [ ] Unit tests cover all validation paths (missing skill, bad hook path, missing actorId, etc.)
- [ ] At least one integration test loads a deliberately broken YAML and asserts the error

Contributor guide

Open the contributing guide

Research direction

Start by reading DomainHarness.__post_init__ and FoundationAgent.__init__, then trace from_yaml(), the registry objects, and the existing HookConfig, PolicyConfig, memory, and evaluation-rule models. Implement validation for each acceptance-criteria path, raise HarnessValidationError during agent construction, and add unit tests plus an integration test for broken YAML with actionable errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
developer-experience
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.