[workshop-sim] Workshop Simulation Report — 2026-09-16 (Run #8, 1000×Monte Carlo)

### Overview
- Date: 2026-09-16
- Students simulated: 46 × 1000 Monte Carlo runs
- Workshop steps available: 30/30
- Overall success rate: 23.0% (95% Monte Carlo interval: 22.7%–23.4%)
- Highest-dropout step: `07-first-workflow` (21.9% conditional dropout among 21,930 at-risk runs; 95% Monte Carlo interval: 21.4%–22.5%)
- Lowest curriculum quality step: `04-github-actions-intro.md` (overall score 5.39/10)
- Learning KPI index: 2.9/10 (active_learning 4.2 · checkpoint_quality 0.0 · scaffolding 5.0)
- Model: `2026-07-survival-model-v2` / `2026-07-assumption-model-v2` (parameter hash `2024391902`)
- Limitation: synthetic results reflect explicit model assumptions; intervals exclude model and population-assumption uncertainty

### Part Summary
| Part | Files | Mean Score | Std Dev |
|---|---|---|---|
| Part 1 — core path (lessons 00–14) | `15` | `6.69 / 10.0` | `±1.74` |
| Part 2 — advanced (lessons 15+) | `15` | `6.06 / 10.0` | `±0.37` |
| Overall corpus | `30` | `6.37 / 10.0` | `±—` |

No steps are classified as `other`.

### Critical Findings
1. **All five highest-dropout steps sit in Part 1 (00–14)**: `07-first-workflow` (21.9%), `05-agentic-intro` (20.0%), `04-actions-intro` (16.2%), `05c-agentic-practice` (12.5%), and `05b-agentic-security` (9.7%). The core onboarding arc — not the advanced Part 2 lessons — is where the cohort loses the most learners.
2. **A simulator lexical-detector gap masked the real signal at Step 8.** The `workflowLockPublishCueCount` regex only matches present-tense `commit`/`push`, but the actual checkpoint text uses `committed`/`pushed`, producing a false ~100% dropout at `08-run-your-workflow` before this run's semantic review. After correcting for the well-instructed publish state, `08-run-your-workflow` drops to a normal 1.4% dropout — the content there is actually strong (browser-first path, explicit lock-file pre-flight checks, linked Codespaces `actions:write` side quest).
3. **`07-first-workflow`'s dominant failure mode is an access barrier, not a learning barrier**: `copilot-access-missing` (3,967 of 4,806 failures) reflects real-world Copilot license/org-billing gaps that content alone cannot fully remediate, even though the billing side quest is thoroughly documented.
4. **Learning quality health is weak across the board**: the Learning KPI index is 2.9/10, driven entirely by `checkpoint_quality = 0.0` on all 30 steps. This is very likely a rubric-detector gap rather than a true absence of checkpoints — every step in the corpus uses the `## :white_check_mark: Checkpoint` shortcode heading, but the shared rubric's `CHECKPOINT_RE` only matches a literal `## ✅ Checkpoint` emoji character, not the `:white_check_mark:` GitHub shortcode. This should be verified against the rubric before trusting `checkpoint_quality` as an improvement target.
5. **The single most important repair belongs to Part 1**: `05-agentic-workflows-intro.md` combines the second-lowest curriculum score (5.43/10) with the second-highest dropout (20.0%) and the lowest `active_learning` score (2.4/10) of any step in the corpus, making it the highest-leverage target for both completion and learning-quality gains.

### Top Repairs to Prioritize
Note: some student dropout is expected and acceptable. Repairs must maintain or improve the learning KPI index — do not lower the cognitive bar or remove practice to chase headline completion numbers.

1. Strengthen recovery support in `05-agentic-workflows-intro.md` for learners who misclassify Activities 2–3 or lack a pre-existing `.lock.yml` to inspect in Activity 1 (completion impact: ↑ · learning KPI impact: ↑ — targets `active_learning` and `scaffolding`, the two weakest dimensions on this page)
2. Reduce concept density and add a worked example in `04-github-actions-intro.md`, the lowest-scoring step in the whole corpus (28 new concepts in 756 words) (completion impact: ↑ · learning KPI impact: ↑ — targets `cognitive_load` and `active_learning`)
3. Add explicit account/org pre-checks earlier in the `07-first-workflow` → `07d-confirm-model-access` flow so Copilot-access gaps surface before the learner invests time authoring a workflow (completion impact: ↑ · learning KPI impact: ↔ — targets `scaffolding`; this is an access-barrier fix and should not touch the existing authoring practice or checklist)

Dropout by step

| Step | At-risk runs | Dropouts | Conditional dropout | 95% MC interval | Failure mode | Top reason |
|---|---|---|---|---|---|---|
| 07-first-workflow | 21,930 | ~4,806 | 21.9% | 21.4%–22.5% | Access barrier | Copilot access/license or org-billing not configured for the account |
| 05-agentic-intro | 37,024 | ~7,407 | 20.0% | 19.6%–20.4% | Learning barrier | Learner misclassifies agentic vs. standard workflow concepts |
| 04-actions-intro | 44,173 | ~7,149 | 16.2% | 15.8%–16.5% | Learning barrier | Concept overload (too many new terms too fast) |
| 05c-agentic-practice | 29,617 | ~3,710 | 12.5% | 12.2%–12.9% | Learning barrier | Learner cannot reliably classify agentic vs. classic scenarios |
| 05b-agentic-security | 25,907 | ~2,508 | 9.7% | 9.3%–10.0% | Learning barrier | Gaps in understanding safe-outputs/sandbox security guardrails |
| 06-install-gh-aw | 23,399 | ~1,469 | 6.3% | 6.0%–6.6% | Access barrier | CLI extension install/auth friction |
| 17-add-mcp-tools | 14,674 | ~701 | 4.8% | 4.4%–5.1% | Learning barrier | MCP tool configuration friction |
| 15-conditional-logic | 15,553 | ~621 | 4.0% | 3.7%–4.3% | Learning barrier | Conditional-logic syntax friction |
| 02-setup | 46,000 | ~1,827 | 4.0% | 3.8%–4.2% | Access barrier | Codespace setup friction |
| 08-run-your-workflow | 17,124 | ~232 | 1.4% | 1.2%–1.5% | Access barrier (minor) | Occasional UI run-guidance gap; corrected after fixing the detector gap noted above |

Curriculum quality and learning KPIs

| Step file | Overall score | active_learning | checkpoint_quality | scaffolding | Learning KPI index | Lowest rubric dimension | Repair priority |
|---|---|---|---|---|---|---|---|
| 04-github-actions-intro.md | 5.39 | 3.9 | 0.0 | 5.0 | 2.78 | checkpoint_quality (likely detector gap) / active_learning | High |
| 05-agentic-workflows-intro.md | 5.43 | 2.4 | 0.0 | 5.0 | 2.24 | active_learning | High |
| 14b-pr-reviewer-workflow.md | 5.45 | 4.3 | 0.0 | 5.0 | 2.62 | checkpoint_quality (likely detector gap) | Medium |
| 08-run-your-workflow.md | 5.67 | 3.0 | 0.0 | 5.0 | 2.44 | active_learning | Medium |
| 05c-agentic-workflows-practice.md | 6.22 | 6.9 | 0.0 | 5.0 | 3.62 | checkpoint_quality (likely detector gap) | Low |
| 07-your-first-workflow.md | 6.25 | 6.3 | 0.0 | 5.0 | 3.65 | checkpoint_quality (likely detector gap) | Low |
| **Cohort mean (30 steps)** | **6.37** | **4.17** | **0.0** | **5.0** | **2.88** | checkpoint_quality (all steps) | — |

Segment breakdowns

**By technical level**
| Level | Mean success rate |
|---|---|
| advanced | 48.7% |
| actions-user | 47.3% |
| github-basic | 15.4% |
| beginner | 0.4% |

**By personality**
| Personality | Mean success rate |
|---|---|
| impatient | 26.2% |
| methodical | 25.1% |
| skeptical | 22.7% |
| confused | 22.1% |
| curious | 20.6% |

**By UI preference**
| ui_preferred | Mean success rate |
|---|---|
| false (CLI-first) | 33.3% |
| true (browser-first) | 11.8% |

Notable student journeys (3)

**Surprising success:** Learner 026 (advanced, devops, confused, CLI, `ui_preferred: false`) reached a 58.6% success rate — the "confused" personality trait mattered far less than expected once paired with advanced-level baseline confidence and a CLI-first workflow that avoids the browser-vs-terminal ambiguity flagged in `07-first-workflow`.

**Unexpected dropout:** Several `beginner`/`no-coding` learners using `tool: CCA` (e.g., Learner 017, Learner 023) hit a 0% success rate, failing overwhelmingly at `04-actions-intro` rather than at the Copilot-specific steps — the GitHub Actions conceptual on-ramp, not the agentic-specific content, is the first wall for true beginners regardless of their preferred tool.

**Content-gap case:** `05-agentic-intro` shows the second-highest dropout (20.0%) driven by `agentic-concept-gap` (5,235 failures) and `deployment-capability-gap` (2,172 failures). Semantic review found the page's three activities are genuinely active (predict-then-reveal, classify, draft-then-compare) but rely entirely on self-graded "reveal answer" boxes with no corrective path for a learner who gets a classification wrong — a real scaffolding gap, not a lexical-detector artifact.

> Generated by [🔬 Workshop Student Simulator](https://github.com/githubnext/gh-aw-workshop/actions/runs/35109821545) · copilot · auto · 164.9 AIC · ⌖ 7.67 AIC · ⊞ 15.2K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-workshop+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-workshop%2Fworkshop-student-simulator%22&type=issues)
> - [x] expires on Sep 17, 2026, 2:52 PM UTC

Start by reading 05-agentic-workflows-intro.md, 04-github-actions-intro.md, and the 07-first-workflow to 07d-confirm-model-access flow, then compare them with the reported failure modes. Review the shared rubric's CHECKPOINT_RE and the Workshop Student Simulator results before changing content. Done means the three prioritized repairs are addressed without lowering practice quality, and the simulator report is rerun to check completion and learning KPIs.

Written by the indexing model from the issue text.