aws-samples / aws-samples/sample-agent-greenhouse
Add fail-fast harness validation at agent construction time
- Dominant language
- Python
- Stars
- 21
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
`DomainHarness.__post_init__` only validates `if not self.name`. There is no validation that:
- Referenced skills exist in the skill registry
- Hook dotted-paths (`HookConfig`) are importable
- Tool allowlists in `PolicyConfig` reference registered tools
- Memory namespace templates contain required placeholders like `{actorId}`
- Evaluation rule scorer references are valid
Misconfigurations in the harness YAML currently surface as **runtime errors deep in the hook chain** — sometimes minutes into an agent session. This results in cryptic `AttributeError` or `ImportError` messages that are difficult to debug.
## Proposed Solution
Add a `validate()` method on `DomainHarness` and call it in `FoundationAgent.__init__`:
```python
@dataclass(frozen=True)
class DomainHarness:
# ... existing fields ...
def validate(self, skill_registry=None, tool_registry=None) -> list[str]:
"""Validate harness configuration. Returns list of error messages."""
errors = []
# 1. Validate skill references
if skill_registry:
for skill_ref in self.skills:
if skill_ref.name not in skill_registry:
errors.append(f"Skill '{skill_ref.name}' not found in registry")
# 2. Validate hook imports
for hook_config in self.hooks:
try:
importlib.import_module(hook_config.module_path)
except ImportError:
errors.append(f"Hook '{hook_config.module_path}' cannot be imported")
# 3. Validate memory namespace templates
if self.memory and self.memory.namespace_template:
if "{actorId}" not in self.memory.namespace_template:
errors.append("Memory namespace_template must contain {actorId}")
# 4. Validate policy tool references
if tool_registry and self.policies:
for tool in self.policies.tool_allowlist:
if tool not in tool_registry:
errors.append(f"Policy references unknown tool '{tool}'")
return errors
```
In `FoundationAgent.__init__`:
```python
errors = harness.validate(skill_registry=self._skill_registry)
if errors:
raise HarnessValidationError(
f"Invalid harness configuration:\n" +
"\n".join(f" - {e}" for e in errors)
)
```
## Impact
This is the **single highest-impact improvement for developer experience**. It turns:
```
AttributeError: 'NoneType' object has no attribute 'execute'
at hooks/tool_policy_hook.py:87 (3 minutes into session)
```
Into:
```
HarnessValidationError: Invalid harness configuration:
- Skill 'nonexistent_skill' not found in registry
- Hook 'platform_agent.hooks.missing' cannot be imported
```
## Acceptance Criteria
- [ ] `DomainHarness.validate()` method checks skills, hooks, tools, memory, and eval rules
- [ ] `FoundationAgent.__init__` calls `validate()` and raises `HarnessValidationError` on failure
- [ ] Error messages are actionable (include the invalid value and what was expected)
- [ ] `from_yaml()` optionally validates during deserialization
- [ ] Unit tests cover all validation paths (missing skill, bad hook path, missing actorId, etc.)
- [ ] At least one integration test loads a deliberately broken YAML and asserts the error
Contributor guide
Research direction
Start by reading DomainHarness.__post_init__ and FoundationAgent.__init__, then trace from_yaml(), the registry objects, and the existing HookConfig, PolicyConfig, memory, and evaluation-rule models. Implement validation for each acceptance-criteria path, raise HarnessValidationError during agent construction, and add unit tests plus an integration test for broken YAML with actionable errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- developer-experience
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100