githubnext / githubnext/gh-aw-workshop

[curriculum-eval] .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py: checkpoint_quality — emoji-shortcode regex misses 88

Closed
#3,440 0 comments 0 reactions 0 assignees View on GitHub
curriculum documentation quality
Dominant language
JavaScript
Stars
47
Forks
19
Avg merge
8h 54m
Merged PRs (30d)
50

Description

**File:** `.github/skills/curriculum-quantitative-assessment/curriculum_assessment.py`
**Overall Score:** N/A (rubric implementation bug affecting all 93 files, corpus mean: `6.14`)

**Flagged Dimensions:**

| Dimension | Score | Benchmark | Delta |
|---|---|---|---|
| checkpoint_quality | 0.0 for 88/93 files | checkpoint present, ≥4 items → up to 10 | −up to 10.0 per file |
| scaffolding | 5.0 for 93/93 files | 10.0 if prereq section present | −5.0 per file where a prereq section actually exists |

**Root Cause (≤ 2 sentences):**
`CHECKPOINT_RE = re.compile(r"##\s+✅\s*Checkpoint", ...)` and the prerequisite-section check `re.search(r"##\s+📋\s*Before You Start", ...)` only match the literal emoji glyphs, but the workshop's actual style convention (documented in `AGENTS.md` and used in 88/93 and 86/93 files respectively) uses GitHub's `:white_check_mark:`/`:clipboard:` shortcode syntax, e.g. `## :white_check_mark: Checkpoint` and `## :clipboard: Before You Start`. The regexes never match these headings, so `has_checkpoint` and `has_prereq_section` are false-negative for nearly the entire corpus.

**Evidence (quoted from the file):**
> `CHECKPOINT_RE = re.compile(r"##\s+✅\s*Checkpoint", re.IGNORECASE)`

Compare against an actual workshop heading (from `workshop/04-github-actions-intro.md`):
> `## :white_check_mark: Checkpoint`

Only 5 files (`side-quest-11-09-agent-session-phases.md`, `side-quest-12-01-iterate-agent-output.md`, `side-quest-13-01-pr-labeler-pattern.md`, `side-quest-13-01-schedule-expressions.md`, `side-quest-13-02-pr-summary-pattern.md`) happen to use the literal `✅` emoji and are the only ones scored correctly; the prereq regex matches **zero** files (`has_prereq_section` is `false` for all 93, even though 86 files have a `## :clipboard: Before You Start` section).

**Learning Science Rationale:**
This is a measurement-validity problem rather than a curriculum-design problem, but it directly undermines the rubric's usefulness: `checkpoint_quality` and `scaffolding` are two of the six weighted dimensions (combined weight 3.5 of 10.0) that the framework uses to operationalize retrieval-practice and advance-organizer principles (Ausubel's advance organizer theory motivates the "Before You Start" scaffolding check; Bjork's testing-effect research motivates the checkpoint-quality check). When the detection regex silently fails, the rubric cannot distinguish a workshop step with excellent checkpoints from one with none — the entire corpus is punished equally, masking real quality differences and producing false "below threshold" flags (e.g. `missing_checkpoint` in 4 of the 5 filed findings below) for pages that actually comply with the style guide.

**Improvement Prompt (for an agent):**
```
In .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py, update the checkpoint and
prerequisite-section detection regexes to match GitHub's emoji shortcode syntax in addition to (or instead
of) literal emoji glyphs:

1. Change CHECKPOINT_RE to match both forms:
CHECKPOINT_RE = re.compile(r"##\s+(?:✅|:white_check_mark:)\s*Checkpoint", re.IGNORECASE)

2. Change the has_prereq_section check to match both forms:
re.search(r"##\s+(?:📋|:clipboard:)\s*Before You Start", raw, re.IGNORECASE)
or re.search(r"##\s+Prerequisites", raw, re.IGNORECASE)

3. Re-run the rubric across the full workshop/ corpus and confirm has_checkpoint and has_prereq_section
now report true for the ~88 and ~86 files respectively that use the shortcode heading style, per
AGENTS.md conventions ("End every workshop step with a `## :white_check_mark: Checkpoint` section").

4. Regenerate corpus-metrics.json, rubric-results.json, and score-history.json and confirm the corpus
mean_score rises to reflect the corrected checkpoint_quality and scaffolding scores.
```

**Expected Score After Fix:** Corpus mean expected to rise from `6.14 / 10.0` to approximately `7.3–7.6 / 10.0` (assuming most detected checkpoints have ≥4 checklist items, since checkpoint_quality is weight 2.0 and scaffolding is weight 1.5, together nearly a third of total weight).

> Generated by [🔬 Curriculum Quality Evaluator](https://github.com/githubnext/gh-aw-workshop/actions/runs/35098365907) · copilot · auto · 88.8 AIC · ⌖ 13.7 AIC · ⊞ 9.5K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-workshop+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-workshop%2Fcurriculum-evaluator%22&type=issues)
> - [x] expires on Sep 17, 2026, 1:01 PM UTC

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py and compare its checkpoint and prerequisite regexes with the shortcode conventions in AGENTS.md and workshop/04-github-actions-intro.md. Run the rubric across the workshop/ corpus, then regenerate corpus-metrics.json, rubric-results.json, and score-history.json. Done means shortcode headings are detected for the reported corpus files and the resulting scores reflect those detections.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.