microsoft / microsoft/aspire.dev

Static eval suite does not run CAT-1/CAT-2 on references/*.md - a large share of Python examples is unchecked

Open
#885 2 comments 0 reactions 1 assignee Claimed by @IEvangelist View on GitHub
Dominant language
MDX
Stars
193
Forks
87
Avg merge
1d 22h
Merged PRs (30d)
73

Description

[`static_checks.py`](https://github.com/microsoft/Dataverse-skills/blob/main/.github/evals/static_checks.py) only loads `skills/*/SKILL.md` (see `skill_files = sorted(skills_dir.glob("*/SKILL.md"))`). It never applies the **Python block validity** (CAT-1) or **auth pattern** (CAT-2) rules to **`skills/*/references/*.md`.

Per Anthropic-style Level 3 design in [`CLAUDE.md`](https://github.com/microsoft/Dataverse-skills/blob/main/CLAUDE.md), long workflows and Python are explicitly pushed into `references/.md`. That means **the more the repo follows its own token-budget guidance, the larger the share of copy-pasted examples that bypass CI**.

## Why this matters

- **Agents load references on demand** for deep work (forms/views, `$apply`, OrgDB settings, multi-table import, etc.). Bad patterns there are as harmful as bad patterns in `SKILL.md`, but **PRs can merge with green “Static skill checks”** while breaking rules the suite is supposed to enforce.
- **Scale:** there are on the order of **~30+** fenced `python` blocks under `references/` vs **~45** in all `SKILL.md` files—a **large fraction** of executable snippets is outside the eval harness.
- **Process failure mode:** reviewers naturally diff `SKILL.md`; reference files are easy to under-review, especially when moving content out of the body for **EVAL-BUDGET-03**.

## What is enforced vs not

| Check | `SKILL.md` | `references/*.md` |
|--------|------------|---------------------|
| EVAL-PY-01 `sys.path.insert` before `from auth import` | Yes | **No** |
| EVAL-PY-04 no all-comment stub blocks | Yes | **No** |
| EVAL-PY-05 no `get_token` in `DataverseClient` blocks | Yes | **No** |
| EVAL-PY-06 `load_env` before `os.environ[...]` (notebook exception) | Yes | **No** |
| EVAL-AUTH-01 no `from scripts.auth import` | Yes | **No** |
| EVAL-AUTH-02 raw HTTP justification | Yes | **No** |
| Frontmatter / Skill boundaries / token budget | Yes | N/A (skip or partial) |

## Suggested fix

1. **Extend the runner** to collect Markdown files under each skill, e.g. `skill_dir / "references" / "*.md"` (and optionally other agent-facing doc paths).
2. For each file, run **`check_python_blocks`**, **`check_auth_patterns`**, and **`check_pac_cli`** (if shell fences appear in references). **Do not** run full CAT-4/CAT-5/CAT-8 on reference files unless you add reference-specific rules.
3. Add a **file label** in failure messages (e.g. `dv-metadata/references/forms-and-views.md python-block-2`) so failures are actionable.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.