Evaluator: expose Fileset-backed skills on Fabric Agent Eval targets

Problem

The Agent Eval SDK already supports injecting one or more agentskills.io bundles into FabricAgentRuntime, including harness routing and per-trial skill provenance. However, the Platform Agent Eval job API cannot use that capability:

• FabricRunnerTarget has no skill input.
• A remotely scheduled job cannot reference a caller-local skill directory.
• Platform Filesets can store the bundle, but Evaluator does not materialize that Fileset into the job runtime.

As a result, SDK-level Fabric Skill Eval works locally, while the same evaluation cannot be expressed through the supported Platform agent-evaluate API.

Proposed behavior

Allow a Fabric target to reference one or more Skill bundles stored in Platform Filesets:

target:
  kind: fabric
  config: ...
  skills:
    - name: lta-analysis
      fileset: default/lta-analysis-skill
      path: skills/lta-analysis

For each configured Skill, the Evaluator job should:

1. Resolve the Fileset in the job workspace, or an explicitly supplied workspace.
2. Download it into job-local ephemeral storage.
3. Resolve and validate a relative bundle path containing SKILL.md.
4. Construct the SDK AgentSkill and pass it to FabricAgentRuntime.
5. Rely on the existing SDK runtime for harness-specific staging, content hashing, and trial provenance.

Requirements

• Preserve existing Fabric target behavior when skills is omitted.
• Support multiple Skills with unique agentskills.io names.
• Reject absolute paths, parent traversal, Windows separators, and post-resolution escapes.
• Reject a selected directory that does not contain SKILL.md.
• Expose the contract in the generated OpenAPI schema.
• Forward the Platform SDK and caller identity used by the job so Files access follows normal authorization.
• Preserve the SDK's Skill name, content hash, adapter, injection mode, and staged location in trial provenance.
• Document the API shape and remote Fileset requirement.

Non-goals

• A first-class Platform Skill catalog or Skill entity.
• Automatic baseline/treated A/B scheduling.
• Native Fabric repeat/attempt semantics.
• Changes to the default CPU task image or Fabric/Relay packaging.
• TestCrew-specific Agent-to-Skill mapping.

Acceptance criteria

1. A Platform Agent Eval job accepts a Fabric target with a Fileset-backed Skill.
2. The job downloads the Fileset and injects the selected bundle into FabricAgentRuntime.
3. Missing Platform SDK access, missing SKILL.md, unsafe paths, and duplicate Skill names fail clearly.
4. Existing Fabric jobs without Skills remain backward compatible.
5. Unit coverage validates materialization and input validation.
6. A live Fabric/Codex job can read the staged Skill and records its provenance in the persisted trial.

Validation evidence

This path has been exercised in a Dev Astra deployment with a real Fabric/Codex Agent Eval job. The agent read a marker Skill from a Platform Fileset, returned the required marker, received exact-match 1.0, and persisted Skill content-hash provenance plus Relay ATIF/ATOF artifacts.

Related SDK foundations: #625, #816, #841, and #844.

Start at the FabricRunnerTarget configuration and the Platform agent-evaluate API contract, then trace how generated OpenAPI schema input reaches FabricAgentRuntime. Review the Fileset materialization and path-validation entry points, plus the unit coverage described in the acceptance criteria. Done means valid remote Filesets inject one or more Skills with provenance, invalid inputs fail clearly, and existing targets without skills still work.