[Feature Request] - Allow For Local Evals
- Dominant language
- TypeScript
- Stars
- 283
- Forks
- 95
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 183
Description
### Description
Currently the agentcore CLI requires a deployed agent to execute evals. Considering the agentcore dev command can spin up the agent locally and stores the telemetry in the .cli folder it would be great if the agentcore CLI supported local evaluation executions.
### Acceptance Criteria
- When an agent and evaluator exists in the agentcore.json a user is able to run evals without first deploying the agent to their AWS account
### Additional Context
_No response_
Contributor guide
Research direction
Start with the agentcore CLI command handling for evals and the agentcore dev command, then inspect how telemetry is stored in the .cli folder and how agentcore.json defines an agent and evaluator. Done means a user can run evals from an agentcore.json containing both without deploying the agent to an AWS account.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- ai, cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100