feat(eval): expose maxTokens / temperature / topP flags for llm-as-a-judge evaluator (Bedrock)
- Lingua principale
- TypeScript
- Stelle
- 290
- Fork
- 96
- Merge medio
- 1g 2h
- PR unite (30g)
- 183
Descrizione
### Summary
The `evaluator llm-as-a-judge create` handler only passes `modelId` to the Bedrock evaluator model config — there is no way to set inference parameters. CloudFormation's [`AWS::BedrockAgentCore::Evaluator InferenceConfiguration`](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-properties-bedrockagentcore-evaluator-inferenceconfiguration.html) supports:
- `MaxTokens` — Integer, minimum `1`
- `Temperature` — Number, `0`–`1`
- `TopP` — Number, `0`–`1`
These map to `BedrockEvaluatorModelConfig.inferenceConfig` and apply to both the Bedrock and Bedrock **Mantle** evaluator paths.
### Current behavior
`src/handlers/eval/evaluator/llm-as-a-judge/create/index.tsx` builds the config with just the model id:
```ts
modelConfig: { bedrockEvaluatorModelConfig: { modelId: flags["model"] } },
```
There are no `--max-tokens`, `--temperature`, or `--top-p` flags, so users get whatever the service/construct defaults are and cannot tune scoring behavior.
For comparison, the **harness** handler already exposes these — see `src/core/project/schema/harness.ts` (`temperature`, `topP`, `maxTokens`) and `src/handlers/harness/parameterHelp.tsx`.
### Requested change
Add flags to `llm-as-a-judge create` (and any `update`/edit path):
- `--max-tokens ` (min 1)
- `--temperature ` (0.0–1.0)
- `--top-p ` (0.0–1.0)
Thread them into `bedrockEvaluatorModelConfig.inferenceConfig` (`maxTokens` / `temperature` / `topP`), omitting any unset value so existing defaults are preserved. Enforce the CFN-documented ranges above.
### Related
Companion construct/schema change: **aws/agentcore-l3-cdk-constructs#315** (the L3 currently hardcodes `{ temperature: 0, maxTokens: 4096 }` for Bedrock and rejects these fields unless the provider is OpenAI). This CLI change depends on that surface being available.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia da src/handlers/eval/evaluator/llm-as-a-judge/create/index.tsx, quindi confronta la gestione dei parametri in src/core/project/schema/harness.ts e src/handlers/harness/parameterHelp.tsx. Individua il corrispondente percorso di aggiornamento o modifica e la relativa validazione; il lavoro è completato quando i tre flag accettano gli intervalli documentati, i valori non impostati vengono omessi e tutti i percorsi dell’evaluator Bedrock ricevono i campi inferenceConfig.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- typescript
- Ambito
- cli
- Tipo di issue
- Funzionalità
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 68/100