githubnext / githubnext/gh-aw-workshop
[curriculum-eval] .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py: checkpoint_quality — emoji-shortcode regex misses 88
- Dominant language
- JavaScript
- Stars
- 47
- Forks
- 19
- Avg merge
- 8h 54m
- Merged PRs (30d)
- 50
Description
**File:** `.github/skills/curriculum-quantitative-assessment/curriculum_assessment.py`
**Overall Score:** N/A (rubric implementation bug affecting all 93 files, corpus mean: `6.14`)
**Flagged Dimensions:**
| Dimension | Score | Benchmark | Delta |
|---|---|---|---|
| checkpoint_quality | 0.0 for 88/93 files | checkpoint present, ≥4 items → up to 10 | −up to 10.0 per file |
| scaffolding | 5.0 for 93/93 files | 10.0 if prereq section present | −5.0 per file where a prereq section actually exists |
**Root Cause (≤ 2 sentences):**
`CHECKPOINT_RE = re.compile(r"##\s+✅\s*Checkpoint", ...)` and the prerequisite-section check `re.search(r"##\s+📋\s*Before You Start", ...)` only match the literal emoji glyphs, but the workshop's actual style convention (documented in `AGENTS.md` and used in 88/93 and 86/93 files respectively) uses GitHub's `:white_check_mark:`/`:clipboard:` shortcode syntax, e.g. `## :white_check_mark: Checkpoint` and `## :clipboard: Before You Start`. The regexes never match these headings, so `has_checkpoint` and `has_prereq_section` are false-negative for nearly the entire corpus.
**Evidence (quoted from the file):**
> `CHECKPOINT_RE = re.compile(r"##\s+✅\s*Checkpoint", re.IGNORECASE)`
Compare against an actual workshop heading (from `workshop/04-github-actions-intro.md`):
> `## :white_check_mark: Checkpoint`
Only 5 files (`side-quest-11-09-agent-session-phases.md`, `side-quest-12-01-iterate-agent-output.md`, `side-quest-13-01-pr-labeler-pattern.md`, `side-quest-13-01-schedule-expressions.md`, `side-quest-13-02-pr-summary-pattern.md`) happen to use the literal `✅` emoji and are the only ones scored correctly; the prereq regex matches **zero** files (`has_prereq_section` is `false` for all 93, even though 86 files have a `## :clipboard: Before You Start` section).
**Learning Science Rationale:**
This is a measurement-validity problem rather than a curriculum-design problem, but it directly undermines the rubric's usefulness: `checkpoint_quality` and `scaffolding` are two of the six weighted dimensions (combined weight 3.5 of 10.0) that the framework uses to operationalize retrieval-practice and advance-organizer principles (Ausubel's advance organizer theory motivates the "Before You Start" scaffolding check; Bjork's testing-effect research motivates the checkpoint-quality check). When the detection regex silently fails, the rubric cannot distinguish a workshop step with excellent checkpoints from one with none — the entire corpus is punished equally, masking real quality differences and producing false "below threshold" flags (e.g. `missing_checkpoint` in 4 of the 5 filed findings below) for pages that actually comply with the style guide.
**Improvement Prompt (for an agent):**
```
In .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py, update the checkpoint and
prerequisite-section detection regexes to match GitHub's emoji shortcode syntax in addition to (or instead
of) literal emoji glyphs:
1. Change CHECKPOINT_RE to match both forms:
CHECKPOINT_RE = re.compile(r"##\s+(?:✅|:white_check_mark:)\s*Checkpoint", re.IGNORECASE)
2. Change the has_prereq_section check to match both forms:
re.search(r"##\s+(?:📋|:clipboard:)\s*Before You Start", raw, re.IGNORECASE)
or re.search(r"##\s+Prerequisites", raw, re.IGNORECASE)
3. Re-run the rubric across the full workshop/ corpus and confirm has_checkpoint and has_prereq_section
now report true for the ~88 and ~86 files respectively that use the shortcode heading style, per
AGENTS.md conventions ("End every workshop step with a `## :white_check_mark: Checkpoint` section").
4. Regenerate corpus-metrics.json, rubric-results.json, and score-history.json and confirm the corpus
mean_score rises to reflect the corrected checkpoint_quality and scaffolding scores.
```
**Expected Score After Fix:** Corpus mean expected to rise from `6.14 / 10.0` to approximately `7.3–7.6 / 10.0` (assuming most detected checkpoints have ≥4 checklist items, since checkpoint_quality is weight 2.0 and scaffolding is weight 1.5, together nearly a third of total weight).
> Generated by [🔬 Curriculum Quality Evaluator](https://github.com/githubnext/gh-aw-workshop/actions/runs/35098365907) · copilot · auto · 88.8 AIC · ⌖ 13.7 AIC · ⊞ 9.5K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-workshop+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-workshop%2Fcurriculum-evaluator%22&type=issues)
> - [x] expires on Sep 17, 2026, 1:01 PM UTC
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py and compare its checkpoint and prerequisite regexes with the shortcode conventions in AGENTS.md and workshop/04-github-actions-intro.md. Run the rubric across the workshop/ corpus, then regenerate corpus-metrics.json, rubric-results.json, and score-history.json. Done means shortcode headings are detected for the reported corpus files and the resulting scores reflect those detections.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100