microsoft / microsoft/wiqd

wiqd agent eval init conflicts with the eval file created by agent scaffolding

Open
#23 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PowerShell
Stars
22
Forks
3
Avg merge
6h 20m
Merged PRs (30d)
4

Description

Observed with WIQD 0.12.2 on Windows.

Problem

A newly scaffolded declarative-agent project already contains evals/prompts.json, but the
documented wiqd agent eval init workflow assumes that it can initialize that file. The two WIQD
workflows therefore conflict on a fresh project.

Steps to reproduce

  1. Scaffold a base declarative-agent project:

    wiqd agent create --name eval-init-test --output ./eval-init-test
    
  2. Enter the generated project directory.

  3. Confirm that evals/prompts.json already exists.

  4. Run:

    wiqd agent eval init --env local
    

Actual behavior

The command exits with code 2:

Eval prompts file already exists at 'evals/prompts.json'. Edit it directly or delete it first.

This happens immediately on a project created by WIQD itself, before the user has initialized or
edited the evaluation suite.

Expected behavior

The scaffold and eval-initialization workflows should compose cleanly. Possible resolutions include:

  • make eval init idempotent and validate or update the scaffolded starter;
  • do not pre-create evals/prompts.json during agent scaffolding;
  • have eval init recognize the scaffolded starter as already initialized and return success with
    clear guidance; or
  • document that eval init is unnecessary for newly scaffolded projects.

Current workaround

Run eval init with an alternate --output path to obtain the current authoritative starter
schema, inspect it, manually edit the original scaffolded evals/prompts.json, and then delete the
alternate file.

Environment

Component Value
WIQD 0.12.2
OS Windows (win32)
Shell PowerShell

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the conflict with wiqd agent create and wiqd agent eval init --env local on Windows, then inspect how scaffolding creates evals/prompts.json and how eval initialization handles an existing file. Compare the generated starter with the authoritative schema from an alternate --output path. Done means a freshly scaffolded project and the documented eval workflow compose cleanly with clear behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
powershell
Domain
cli, devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.