QuantEcon / QuantEcon/quantecon-book-theme

ci: add nightly integration smoke test against lecture-python-programming

Open
#393 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
56
Forks
24
Avg merge
4h 33m
Merged PRs (30d)
4

Description

Context

PR #390 migrated visual regression testing from `lecture-python-programming` to the curated `quantecon-book-theme-fixtures` repo, pinning fixtures + their build dependencies (jupyter-book, sphinx, extensions) to a specific SHA + version set. This was the right call for stability — visual diffs now attribute cleanly to theme changes.

However, this introduces a new blind spot: version drift between the pinned fixtures stack and what real QuantEcon lectures actually use today. If jupyter-book or sphinx ships a backwards-incompatible change, our CI won't notice because the fixtures repo stays on the old pinned version — but real lecture sites consuming this theme will break.

The existing `lecture-python-programming` build was useful as a real-world canary for this kind of drift. We removed it from the per-PR critical path (too slow, too noisy) but should bring it back as a non-blocking nightly smoke test.

Proposed solution

Add `.github/workflows/integration.yml` — runs nightly on a cron, builds `lecture-python-programming` from scratch with the latest released theme, fails loudly if the build breaks but does not gate any PR.

Sketch
name: Integration Smoke Test (lecture-python-programming)
on:
  schedule:
    - cron: \"0 6 * * *\"      # daily at 06:00 UTC
  workflow_dispatch:

jobs:
  smoke:
    runs-on: ubuntu-latest
    continue-on-error: true   # explicitly non-blocking
    steps:
      - uses: actions/checkout@v6

      - name: Fetch lecture-python-programming
        run: git clone --branch quantecon-book-theme https://github.com/QuantEcon/lecture-python-programming

      - name: Setup Anaconda
        uses: conda-incubator/setup-miniconda@v4
        with:
          auto-update-conda: true
          python-version: \"3.13\"
          environment-file: lecture-python-programming/environment.yml
          activate-environment: lecture-python-programming

      - name: Install theme from main
        shell: bash -l {0}
        run: pip install .

      - name: Build HTML
        shell: bash -l {0}
        run: |
          cd lecture-python-programming
          jb build lectures --path-output ./

      # Optional: open an issue if the build fails, so we don't have to
      # manually scan failed workflow runs
      - name: Open issue on failure
        if: failure()
        uses: actions/github-script@v9
        with:
          script: |
            const date = new Date().toISOString().split('T')[0];
            await github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: \`Integration smoke failure: \${date}\`,
              body: \`Nightly lecture-python-programming build broke on \${date}.\\n\\nWorkflow run: \${context.serverUrl}/\${context.repo.owner}/\${context.repo.repo}/actions/runs/\${context.runId}\\n\\nLikely cause: jupyter-book / sphinx / extension version drift. Investigate and either bump the fixtures pin in \\\`quantecon-book-theme-fixtures\\\` or fix the theme to match.\`,
              labels: ['ci', 'integration-smoke']
            });

Why this matters

Without this:

  • A jb/sphinx version bump that breaks the theme could ship to PyPI and only get noticed when a lecture maintainer reports a broken build
  • The fixtures pin would mask the issue indefinitely on PR CI
  • Drift accumulates silently between releases

With this:

  • Nightly heartbeat against the real, current lecture-python-programming environment
  • Failures are visible (via the auto-opened issue) without spamming PR reviewers
  • Fix-forward path is clear: either bump `FIXTURES_SHA` in this repo (after refreshing the fixtures repo), or patch the theme to handle the new jb/sphinx version

Tradeoffs

Pros:

  • Closes the drift blind spot introduced by the fixtures migration
  • Non-blocking — doesn't slow down or destabilize PR workflow
  • Cheap: one nightly run ≈ 6 min of CI time
  • Self-documenting failure mode via auto-opened issue

Cons:

  • Bot-opened issues can be noisy if drift is frequent (mitigation: dedupe by checking for an existing open `integration-smoke` issue before creating a new one)
  • Requires occasional manual triage to bump fixtures or patch the theme
  • Conda + jb build is slow; can't avoid this since we're explicitly testing the full real environment

Acceptance criteria

  • `.github/workflows/integration.yml` exists and runs nightly
  • Workflow uses `continue-on-error: true` at the job level so failures don't show as a red ✗ on the repo's main branch status
  • Failure auto-opens an issue labelled `ci` + `integration-smoke` (or comments on an existing open one rather than spamming)
  • `workflow_dispatch` trigger included for manual investigation
  • Documented in `docs/developer/visual-testing.md` (the "Relationship to lecture-python-programming" section) and in `quantecon-book-theme-fixtures/MIGRATION.md`

References

  • #390 — fixtures-based visual regression migration that introduced this gap
  • `MIGRATION.md` — the spec mentions this follow-up
  • `docs/developer/visual-testing.md` — current doc references the fixtures-only flow; needs an "Integration smoke" subsection once this lands

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing .github/workflows/ files, docs/developer/visual-testing.md, and the referenced quantecon-book-theme-fixtures/MIGRATION.md. Add .github/workflows/integration.yml with the nightly and manual triggers, non-blocking lecture build, and failure handling described in the issue. Done means the workflow runs successfully or reports failures without gating PRs, and both documentation locations describe the integration smoke test.

Written by the indexing model from the issue text.

Assessment

Tech stack
anaconda, github-actions, jupyter, python
Domain
build-system, ci-cd, documentation, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.