microsoft / microsoft/Dataverse-skills
Static eval suite does not run CAT-1/CAT-2 on `references/*.md`
- Dominant language
- No language data
- Stars
- 226
- Forks
- 61
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 7
Description
**Type:** Bug / tooling gap
**Area:** `.github/evals/static_checks.py`, skill references
---
## Summary
[`static_checks.py`](https://github.com/microsoft/Dataverse-skills/blob/main/.github/evals/static_checks.py) only loads `skills/*/SKILL.md` (see `skill_files = sorted(skills_dir.glob("*/SKILL.md"))`). It never applies the **Python block validity** (CAT-1) or **auth pattern** (CAT-2) rules to **`skills/*/references/*.md`.
Per Anthropic-style Level 3 design in [`CLAUDE.md`](https://github.com/microsoft/Dataverse-skills/blob/main/CLAUDE.md), long workflows and Python are explicitly pushed into `references/.md`. That means **the more the repo follows its own token-budget guidance, the larger the share of copy-pasted examples that bypass CI**.
---
## Why this matters
- **Agents load references on demand** for deep work (forms/views, `$apply`, OrgDB settings, multi-table import, etc.). Bad patterns there are as harmful as bad patterns in `SKILL.md`, but **PRs can merge with green “Static skill checks”** while breaking rules the suite is supposed to enforce.
- **Scale:** there are on the order of **~30+** fenced `python` blocks under `references/` vs **~45** in all `SKILL.md` files—a **large fraction** of executable snippets is outside the eval harness.
- **Process failure mode:** reviewers naturally diff `SKILL.md`; reference files are easy to under-review, especially when moving content out of the body for **EVAL-BUDGET-03**.
---
## What is enforced vs not
| Check | `SKILL.md` | `references/*.md` |
|--------|------------|---------------------|
| EVAL-PY-01 `sys.path.insert` before `from auth import` | Yes | **No** |
| EVAL-PY-04 no all-comment stub blocks | Yes | **No** |
| EVAL-PY-05 no `get_token` in `DataverseClient` blocks | Yes | **No** |
| EVAL-PY-06 `load_env` before `os.environ[...]` (notebook exception) | Yes | **No** |
| EVAL-AUTH-01 no `from scripts.auth import` | Yes | **No** |
| EVAL-AUTH-02 raw HTTP justification | Yes | **No** |
| Frontmatter / Skill boundaries / token budget | Yes | N/A (skip or partial) |
---
## Suggested fix
1. **Extend the runner** to collect Markdown files under each skill, e.g. `skill_dir / "references" / "*.md"` (and optionally other agent-facing doc paths).
2. For each file, run **`check_python_blocks`**, **`check_auth_patterns`**, and **`check_pac_cli`** (if shell fences appear in references). **Do not** run full CAT-4/CAT-5/CAT-8 on reference files unless you add reference-specific rules.
3. Add a **file label** in failure messages (e.g. `dv-metadata/references/forms-and-views.md python-block-2`) so failures are actionable.
Contributor guide
Research direction
Start in .github/evals/static_checks.py, reading how skill_files are collected and how check_python_blocks, check_auth_patterns, and check_pac_cli report failures. Add coverage for Markdown files under each skill's references directory while preserving the stated CAT-4/CAT-5/CAT-8 exclusions, and include each source file in failure labels. Verify the static eval suite passes and reports actionable reference-file locations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100