runevals: support file_urls on eval items so file-grounded agent behavior can be tested
Nobody has claimed this yet.
- Dominant language
- PowerShell
- Stars
- 22
- Forks
- 3
- Avg merge
- 6h 20m
- Merged PRs (30d)
- 4
Description
Summary
runevals cannot exercise any agent behavior that depends on file context, because its A2A client only ever sends a single text part.
Evidence
In src/clients/cli/api_clients/A2A/a2a_client.py (_build_chat_payload, ~line 339) the payload is hardcoded:
"parts": [{"kind": "text", "text": prompt}],
and send_prompt(prompt: str, ...) accepts only a string. No FilePart/DataPart is ever constructed; every other parts reference in that file is response parsing. The prompts schema has no attachment/file field either.
By contrast, the Work IQ CLI already supports this:
workiq ask -f, --file-urls <file-urls> OneDrive or SharePoint file URLs to use as context
Verified working: passing a OneDrive link to a deployed declarative agent via workiq ask --file-urls let the agent retrieve and review the shared package. The same link embedded in the prompt text (the only option in runevals) fails to resolve, so the agent correctly falls back to "I could not retrieve that file".
Impact
Any file-grounded scenario is untestable in an automated suite: document grounding, attachment analysis, and declarative-agent package review. Worse, such evals appear to run: the agent responds gracefully to the retrieval failure, so the item can PASS while never exercising the intended behavior. That is a silent false negative in regression coverage.
Request
Add an optional file_urls (array of strings) field to eval items and to turns entries in the prompts schema, and plumb it through send_prompt into _build_chat_payload using the same mechanism as workiq ask --file-urls.
Environment
- @microsoft/m365-copilot-eval as installed by wiqd
- workiq CLI
- Scenario: declarative agent that reviews DA packages shared via OneDrive/SharePoint links
Environment Context (auto-collected by wiqd CLI)
| Property | Value |
|---|---|
| wiqd version | 0.10.0 |
| OS | win32 |
| Node.js | v24.18.0 |
| gh CLI | gh version 2.96.0 (2026-07-02) |
| Shell | PowerShell |
| Terminal | Windows Terminal |
| Agent project | yes |
| Submitted at | 2026-08-27T14:55:53.503Z |
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/clients/cli/api_clients/A2A/a2a_client.py at send_prompt and _build_chat_payload, then trace how eval items and turns are represented in the prompts schema. Follow the existing Work IQ CLI --file-urls behavior and confirm file URLs become the appropriate A2A parts while preserving text prompts. Done means file-grounded eval items and turns can pass file_urls through and exercise the intended agent behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100