microsoft / microsoft/wiqd

runevals: support file_urls on eval items so file-grounded agent behavior can be tested

Open
#35 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
PowerShell
Stars
22
Forks
3
Avg merge
6h 20m
Merged PRs (30d)
4

Description

Summary

runevals cannot exercise any agent behavior that depends on file context, because its A2A client only ever sends a single text part.

Evidence

In src/clients/cli/api_clients/A2A/a2a_client.py (_build_chat_payload, ~line 339) the payload is hardcoded:

"parts": [{"kind": "text", "text": prompt}],

and send_prompt(prompt: str, ...) accepts only a string. No FilePart/DataPart is ever constructed; every other parts reference in that file is response parsing. The prompts schema has no attachment/file field either.

By contrast, the Work IQ CLI already supports this:

workiq ask -f, --file-urls <file-urls>   OneDrive or SharePoint file URLs to use as context

Verified working: passing a OneDrive link to a deployed declarative agent via workiq ask --file-urls let the agent retrieve and review the shared package. The same link embedded in the prompt text (the only option in runevals) fails to resolve, so the agent correctly falls back to "I could not retrieve that file".

Impact

Any file-grounded scenario is untestable in an automated suite: document grounding, attachment analysis, and declarative-agent package review. Worse, such evals appear to run: the agent responds gracefully to the retrieval failure, so the item can PASS while never exercising the intended behavior. That is a silent false negative in regression coverage.

Request

Add an optional file_urls (array of strings) field to eval items and to turns entries in the prompts schema, and plumb it through send_prompt into _build_chat_payload using the same mechanism as workiq ask --file-urls.

Environment

  • @microsoft/m365-copilot-eval as installed by wiqd
  • workiq CLI
  • Scenario: declarative agent that reviews DA packages shared via OneDrive/SharePoint links

Environment Context (auto-collected by wiqd CLI)
Property Value
wiqd version 0.10.0
OS win32
Node.js v24.18.0
gh CLI gh version 2.96.0 (2026-07-02)
Shell PowerShell
Terminal Windows Terminal
Agent project yes
Submitted at 2026-08-27T14:55:53.503Z

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/clients/cli/api_clients/A2A/a2a_client.py at send_prompt and _build_chat_payload, then trace how eval items and turns are represented in the prompts schema. Follow the existing Work IQ CLI --file-urls behavior and confirm file URLs become the appropriate A2A parts while preserving text prompts. Done means file-grounded eval items and turns can pass file_urls through and exercise the intended agent behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.