GitHub Actions Workflows

GitHub Actions Workflows#

This page documents QuantEcon’s GitHub Actions workflow configurations.

Documentation Preview Workflow#

The QuantEcon.manual repository includes an automated CI workflow that provides documentation previews for Pull Requests.

How It Works#

When you create or update a Pull Request, the ci.yml workflow automatically:

  1. Builds the documentation using Jupyter Book with the same environment as production

  2. Deploys a preview to https://manual.quantecon.org/pr-{number}/

  3. Comments on the PR with a direct link to the preview

  4. Updates the preview when you push new commits to the PR

  5. Cleans up the preview directory when the PR is closed or merged

The cleanup process includes:

  • Automatic Detection: Distinguishes between merged and closed PRs

  • Serialised gh-pages writes: All jobs that push to gh-pages (preview deploys, cleanup, and the main publish) share one concurrency group, so a concurrent deploy cannot resurrect a deleted preview

  • Verification: Fetches origin/gh-pages after pushing and fails the job if the preview directory is still present

  • Error Handling: Gracefully handles cases where directories are already removed

Benefits#

  • Visual Feedback: See exactly how your changes will look before merging

  • Review Process: Reviewers can easily access the rendered documentation

  • Quality Assurance: Catch formatting issues and broken links early

  • No Manual Setup: Everything is automated - just create a PR

Preview URLs#

Previews are accessible at:

  • Primary: https://manual.quantecon.org/pr-{number}/

  • Fallback: https://quantecon.github.io/QuantEcon.manual/pr-{number}/

Where {number} is your PR number (e.g., pr-45 for Pull Request #45).

Workflow Configuration#

The CI workflow is defined in .github/workflows/ci.yml and includes:

  • Environment Setup: Uses the same conda environment as production (environment.yml)

  • Build Process: Runs jb build manual --path-output ./

  • Safe Deployment: Uses subdirectory deployment to avoid conflicts with the main site

  • Smart Comments: Creates new preview comments for each build with commit SHA

  • Robust Cleanup: Removes preview directories when PRs are closed with:

    • Automatic detection of merged vs. closed PRs

    • Rebase-and-retry logic for push operations, failing the job if the push never lands

    • Verification against origin/gh-pages (not the local working tree)

    • Graceful handling of edge cases

Note

The preview workflow is completely safe and won’t interfere with the main documentation site. It uses subdirectory deployment and preserves all existing content and configuration.

When the main branch publishes to gh-pages, the publish.yml workflow is configured with keep_files: true to preserve existing PR preview directories, so active previews remain accessible even after main branch deployments.

Because peaceiris/actions-gh-pages pushes from a snapshot of gh-pages taken when its job starts, all jobs that write to the branch are serialised through a shared gh-pages concurrency group — without it, overlapping runs silently undo each other’s commits (this is how deleted previews were resurrected; see issue #121).

GPU-Enabled Workflows#

For lecture repositories that require GPU computation, QuantEcon uses specialized workflows.

Our GPU based workflows are now built using RunsOn

Build Images#

There are two types of workflows:

  1. Using our own custom ami that is preloaded with CUDA, CUDANN, and LaTeX

  2. Using the ami provided by Runs On that includes Docker support. This is used by collab.yml workflows or any containerized workflow.

The benefit of using our own custom AMI is that we can choose CUDA, CUDANN versions and ensure we are able to use the latest versions to support lectures on JAX and GPU computing.

It is best to install only system wide software such as GPU frameworks, tool-kits and drivers.

When setting up a custom AMI we should install the NVIDIA support:

  1. nvidia CUDA framework

  2. nvidia drivers

  3. nvidia CUDANN

using the latest versions and then LaTeX

sudo apt-get -qq update
sudo apt-get install -y     \
texlive-latex-recommended \
texlive-latex-extra       \
texlive-fonts-recommended \
texlive-fonts-extra       \
texlive-xetex             \
latexmk                   \
xindy                     \
dvipng                    \
cm-super

Leave anaconda and local installs to the GitHub actions workflow.

While this incurs a small setup cost in terms of execution time, it provides flexibility in upgrading the python / anaconda software stack.

In addition to anaconda, you should install GPU-enabled Python packages in the GitHub Action as a setup step. The current standard across the GPU lecture repositories (e.g. lecture-jax, lecture-python-programming) is:

  - name: Install JAX and Numpyro
    shell: bash -l {0}
    run: |
        pip install -U "jax[cuda13]"
        pip install numpyro
        python scripts/test-jax-install.py

Note

The jax[cuda13] selector installs JAX against the CUDA 13 wheels and is matched to the CUDA / driver version baked into the custom AMI (see below). Bump this selector when the AMI is rebuilt against a newer CUDA release.

If a lecture also requires PyTorch / Pyro, add them as a separate install step:

  - name: Install PyTorch and Pyro
    shell: bash -l {0}
    run: |
        pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu130
        pip install pyro-ppl

Warning

PyTorch typically lags JAX in terms of compatibility with NVIDIA drivers. It is best to install the nightly versions to ensure the best compatibility, and to match the CUDA index URL (cu130, cu13x, …) to the CUDA version on the AMI.

You can check the latest on the PyTorch site with the installation table

Standard Lecture Workflows#

A typical QuantEcon lecture repository (e.g. lecture-python.myst) ships the following workflows in .github/workflows/:

Workflow

Trigger

Purpose

ci.yml

Pull requests

Build the lecture and generate a preview for review

cache.yml

Push to main (and schedule)

Execute the full project and save the _build execution cache as a workflow artifact

publish.yml

Push of a publish* tag (legacy flow)

Build and deploy the live site (see Publishing Lectures). Repositories migrated to quantecon/actions publish on push to main instead — see Templates

linkcheck.yml

Schedule / manual

Validate internal and external links

collab.yml

Manual / containerized

Build inside a Docker container on a RunsOn runner (used for collab and container workflows)

PR and publish builds restore the _build artifact produced by cache.yml, so unchanged lectures are not re-executed. See Setup Cache for GitHub Actions for the configuration detail.

QuantEcon Actions — Reusable Build & Publish Library#

Note

This is the target architecture for QuantEcon lecture CI/CD. New repositories should adopt it and existing repositories are being migrated over time. See the lecture deployment table for the current per-repo status.

The QuantEcon/actions repository provides a set of reusable composite GitHub Actions that standardise and speed up lecture builds. Rather than each repository carrying a long, hand-maintained publish.yml, the build is composed from small, independently versioned actions (released on the v0 tag line — see releases for the current version). lecture-dp is the reference implementation.

The design rests on three pillars:

  1. Containerised environment — a pre-built Docker image ghcr.io/quantecon/quantecon-build:latest (Ubuntu 24.04 + TexLive + Miniconda + Anaconda 2025.12 base + Jupyter Book tooling), rebuilt weekly and published to the GitHub Container Registry. This removes the 2–3 minute LaTeX install from every run. The container is built from the containers/ definitions in the same repo.

  2. Modular actions — small composable steps with clear responsibilities (see table below).

  3. Two-layer caching — the container image (environment) plus the Jupyter _build cache (execution), so only changed lectures are re-executed.

How the build environment is resolved#

Which file governs a repository’s build depends on the mode setup-environment runs in:

  • Container mode — when the job runs inside ghcr.io/quantecon/quantecon-build (or the full ghcr.io/quantecon/quantecon image), the container supplies the base environment. If the workflow passes an environment-update: input, the named file is applied as a delta on top via conda env update (~30–60 seconds); if not, the pre-installed packages are used as-is and the repository’s environment.yml is not consulted.

  • Standard mode — on ubuntu-latest or any non-container runner, the full conda environment is created from the repository’s environment.yml (cached on its hash), with optional LaTeX install.

So under the container-first architecture a repository’s environment file is either a delta or a full specification — to know which file governs a given build, check whether the workflow sets container: and which input (environment-update: vs environment:) it passes. The container definitions live in containers/, and the setup-environment README documents the full input set.

Available actions#

Action

Role

setup-environment

Flexible environment setup (optional Conda / LaTeX / ML libs); container-aware

build-lectures

Build Jupyter Book (HTML, PDF, notebooks) with unified error handling and asset assembly

build-jupyter-cache

Weekly execution-cache generation on main (validates all builders pass before saving)

restore-jupyter-cache

Read-only cache restore for PR builds (never writes, so PRs can’t corrupt the cache)

publish-gh-pages

Deploy to GitHub Pages using native artifact deployment (no gh-pages branch); optional custom domain and release assets

preview-netlify

Deploy PR previews to Netlify with smart PR comments

preview-cloudflare

Deploy PR previews to Cloudflare Pages (pr-N.project.pages.dev)

Important

publish-gh-pages deploys via the native GitHub Pages artifact pipeline (actions/deploy-pages), so repositories on this path do not have a gh-pages branch. The local ghp-import fallback documented in Building Locally and Push to GitHub Pages does not apply to them.

Minimal publish workflow#

A migrated publish.yml reduces to roughly:

jobs:
  build:
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/quantecon/quantecon-build:latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: quantecon/actions/restore-jupyter-cache@v0
      - uses: quantecon/actions/build-lectures@v0
        id: build
        with:
          builder: 'html'
  deploy:
    needs: build
    runs-on: ubuntu-latest
    permissions: { contents: write, pages: write, id-token: write }
    steps:
      - uses: quantecon/actions/publish-gh-pages@v0
        with:
          build-dir: _build/html
          github-token: ${{ secrets.GITHUB_TOKEN }}

See also

The migration guide and architecture overview in the quantecon/actions repository document the full set of inputs and the step-by-step process for converting a repository.

Repository Backup Workflow#

QuantEcon uses an automated backup system to protect repositories against data loss. The backup workflow is managed from the workflow-backups repository.

Overview#

The backup workflow:

  1. Runs weekly (Sunday at 2 AM UTC) via GitHub Actions

  2. Creates mirror backups of selected repositories (all branches, tags, and history)

  3. Stores backups securely in AWS S3

  4. Generates reports on backup status and storage usage

Repository Selection#

Repositories are selected for backup using pattern-based matching in config.yml:

backup:
  enabled: true
  organization: "QuantEcon"
  
  # Exact repository names
  repositories:
    - "QuantEcon.py"
    - "QuantEcon.jl"
  
  # Regex patterns for matching multiple repos
  patterns:
    - "lecture-.*"      # All lecture repositories
    - "quantecon-.*"    # All quantecon packages
  
  s3:
    bucket: "your-backup-bucket"
    prefix: "backups/"

Authentication Setup#

The backup workflow requires two types of authentication:

AWS Authentication (OIDC)#

AWS access uses OIDC authentication (no long-lived credentials):

  1. An IAM Identity Provider is configured for GitHub Actions

  2. An IAM Role with S3 permissions is created

  3. The role ARN is stored as a repository secret: AWS_ROLE_ARN

GitHub Token for Private Repositories#

Important

Fine-Grained Personal Access Token (PAT) Setup

For backing up private repositories, a Fine-Grained PAT must be created with READ-ONLY permissions:

  1. Go to GitHub Settings → Developer settings → Personal access tokens → Fine-grained tokens

  2. Click “Generate new token”

  3. Set the following:

    • Token name: quantecon-backup-readonly (or similar descriptive name)

    • Expiration: Set an appropriate expiration (recommend 90 days or custom)

    • Resource owner: Select the QuantEcon organization

    • Repository access: Select “All repositories” or specific repositories to backup

    • Permissions: Set Repository permissions to:

      • Contents: Read-only

      • Metadata: Read-only (automatically selected)

    • All other permissions should remain at “No access”

  4. Generate the token and store it as the REPO_BACKUP_TOKEN secret in the workflow-backups repository

Security Note: The token should have minimal READ-ONLY permissions. Never grant write access for backup operations.

For public repositories only, the default GITHUB_TOKEN works automatically with no additional setup.

Backup Storage Structure#

Backups are stored in S3 with the following structure:

s3://bucket-name/
├── lecture-python/
│   ├── lecture-python-20251127.tar.gz
│   ├── lecture-python-20251120.tar.gz
│   └── lecture-python-20251113.tar.gz
└── QuantEcon.py/
    └── QuantEcon.py-20251127.tar.gz

Each backup is a complete git mirror including all branches, tags, and full commit history.

Restoring from Backup#

To restore a repository from backup:

# 1. Download the backup from S3
aws s3 cp s3://bucket-name/repo-name/repo-name-20251127.tar.gz .

# 2. Extract the archive (contains a bare git mirror)
tar -xzf repo-name-20251127.tar.gz

# 3. Clone from the bare repo to create a working repository
git clone repo-name restored-repo

# 4. Verify the restoration
cd restored-repo
git branch -a   # View all branches
git tag         # View all tags

Manual Trigger#

The backup workflow can be triggered manually via GitHub Actions:

  1. Go to the workflow-backups Actions page

  2. Select the “Repository Backup” workflow

  3. Click “Run workflow”

More Information#

For detailed setup instructions and configuration options, see the workflow-backups README.

Activity & News Reports#

QuantEcon runs scheduled workflows that summarise organization activity and draft content for review — weekly/monthly status reports and draft news posts for quantecon.org. These are semi-automated (human-in-the-loop) and managed in the reports-activity repository. See Activity Reports for details.