dotnet / dotnet/skills

dotnet-aot-compat: evaluation scenario too large to complete within timeout

Open
#367 0 comments 1 reaction 2 assignees Claimed by @danmoseley View on GitHub
area-upgrade
Dominant language
C#
Stars
5.4k
Forks
415
Avg merge
1d 4h
Merged PRs (30d)
84

Description

The `dotnet-aot-compat` evaluation scenario (Azure.ResourceManager — 461 files, 135+ JSON serialization call sites) consistently hits the 1050s timeout in both baseline and withSkill runs. This means:

1. **We can't measure skill quality** — every run times out before the agent finishes, so judge scores reflect "how far did it get" rather than "did the approach work." Per-run variance is huge (improvement scores ranged from -0.20 to +0.22 across runs of the same wording).

2. **Real users likely hit the same wall** — a user with Copilot working on a similarly large codebase will get an incomplete result regardless of how good the skill guidance is.

### Proposed follow-ups

#### 1. Use a smaller test project for evaluation

Replace or supplement Azure.ResourceManager with a project that has ~20-30 AOT warning sites instead of 135+. This lets the agent actually complete the task within the timeout, making skill improvements measurable. The current scenario is useful as a stress test but not as a quality signal.

#### 2. Add scope-limiting guidance to the skill

When the codebase has a very large number of warnings, the skill should guide the agent to prioritize rather than attempt to fix everything. For example:
- "If there are more than ~30 distinct warning sites, focus on the highest-impact patterns first (e.g., JSON serialization, reflection-based factories) and document remaining warnings as TODOs"
- This helps both evaluation (agent can finish in time) and real users (they get a useful partial result instead of a timed-out incomplete one)

### Context

This was discovered during analysis of #366 evaluation results. See the [detailed comparison comment](https://github.com/dotnet/skills/pull/366#issuecomment-4059507897). The batching guidance in #366 is a step in the right direction (judge confirmed the agent actually attempted builds with new wording vs. zero builds with old), but the timeout is the dominant factor masking any improvement.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.