aws / aws/agentcore-cli

[Feature Request] - Allow For Local Evals

Open
#2,047 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
283
Forks
95
Avg merge
1d 2h
Merged PRs (30d)
183

Description

### Description

Currently the agentcore CLI requires a deployed agent to execute evals. Considering the agentcore dev command can spin up the agent locally and stores the telemetry in the .cli folder it would be great if the agentcore CLI supported local evaluation executions.

### Acceptance Criteria

- When an agent and evaluator exists in the agentcore.json a user is able to run evals without first deploying the agent to their AWS account

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the agentcore CLI command handling for evals and the agentcore dev command, then inspect how telemetry is stored in the .cli folder and how agentcore.json defines an agent and evaluator. Done means a user can run evals from an agentcore.json containing both without deploying the agent to an AWS account.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
ai, cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.