aws / aws/nova-prompt-optimizer

Add support for agentic workflow optimization and cost-aware model routing

Open
#28 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
56
Forks
9
PR merge metrics
No merged PRs in 30d

Description

# Feature Request: Agentic workflow optimization support

## Problem
Current optimization targets single prompts, but modern LLM apps use multi-step agent workflows. These have different challenges:
- Agents make 10-100x more API calls than single prompts
- Need to optimize entire workflows, not just individual steps
- Different models work better for different workflow steps
- Hard to evaluate multi-step success vs. individual responses

## Proposed Solution
Add workflow optimization that can optimize multi-step agent pipelines with cost-aware model routing.

### Basic Usage
```python
# Define multi-step workflow
workflow = AgentWorkflow([
PlanningStep(models=["nova-pro", "claude-sonnet"]),
ReasoningStep(models=["claude-sonnet", "gpt-4o"]),
ToolCallingStep(models=["gpt-4o", "nova-lite"]),
SynthesisStep(models=["nova-pro"])
])

# Optimize entire workflow
optimizer = WorkflowOptimizer(
workflow=workflow,
cost_budget=50.0,
metric=workflow_success_metric
)

optimized_workflow = optimizer.optimize(dataset, metric)
```

### Cost-Aware Model Routing
```python
# Automatically route based on task complexity and cost
router = CostAwareRouter({
"simple_tasks": "nova-lite", # $0.0006/1k tokens
"reasoning": "claude-sonnet", # $0.003/1k tokens
"complex_coding": "gpt-4o" # $0.005/1k tokens
})

workflow.add_router(router)
```

## Key Features Needed

- **Workflow adapter**: Execute and track multi-step workflows
- **Step-level optimization**: Optimize prompts for each workflow step
- **Model routing**: Assign optimal models to different steps
- **Workflow metrics**: Evaluate entire pipeline success
- **Cost tracking**: Monitor costs across all workflow steps
- **Context management**: Optimize data passing between steps

## Example Use Cases

**Research Agent**: Search (nova-lite) → Analysis (claude-sonnet) → Synthesis (nova-pro)
**Coding Agent**: Planning (claude-sonnet) → Implementation (gpt-4o) → Testing (nova-pro)
**Support Agent**: Classification (nova-lite) → Retrieval (nova-pro) → Response (claude-sonnet)

## Why This Matters

1. **Agent workflows are becoming standard** - single prompts are less common/for prototypes
2. **Cost explosion problem** - agents can easily burn through budgets
3. **No existing tools** optimize multi-step workflows end-to-end
4. **Model specialization** - different models excel at different workflow steps

## Consider
- Extend existing `PromptAdapter` to support workflow definitions
- Add `WorkflowOptimizer` that uses MIPROv2 across multiple steps
- Create workflow-specific metrics and evaluation methods
- Add cost tracking and budget allocation across steps

## References
- [LangGraph](https://github.com/langchain-ai/langgraph) - Agent workflow patterns
- [DSPy ReAct](https://dspy-docs.vercel.app/docs/building-blocks/modules#dspyreact) - Single-step agent optimization
- [CrewAI](https://github.com/joaomdmoura/crewAI) - Multi-agent frameworks

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing PromptAdapter and the proposed MIPROv2 extension points. Then map how workflow steps, model routing, metrics, cost tracking, and context passing would fit together. Done would require an end-to-end workflow optimizer with step-level model selection, cost limits, and evaluation of whole-workflow success.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.