GitHub Actions Workflows

GitHub Actions Workflows#

This page documents QuantEcon’s GitHub Actions workflow configurations.

Documentation Preview Workflow#

The QuantEcon.manual repository includes an automated CI workflow that provides documentation previews for Pull Requests.

How It Works#

When you create or update a Pull Request, the ci.yml workflow automatically:

  1. Builds the documentation using Jupyter Book with the same environment as production

  2. Deploys a preview to https://manual.quantecon.org/pr-{number}/

  3. Comments on the PR with a direct link to the preview

  4. Updates the preview when you push new commits to the PR

  5. Cleans up the preview directory when the PR is closed or merged

The cleanup process includes:

  • Automatic Detection: Distinguishes between merged and closed PRs

  • Robust Removal: Safely removes preview directories with retry logic

  • Verification: Confirms successful cleanup completion

  • Error Handling: Gracefully handles cases where directories are already removed

Benefits#

  • Visual Feedback: See exactly how your changes will look before merging

  • Review Process: Reviewers can easily access the rendered documentation

  • Quality Assurance: Catch formatting issues and broken links early

  • No Manual Setup: Everything is automated - just create a PR

Preview URLs#

Previews are accessible at:

  • Primary: https://manual.quantecon.org/pr-{number}/

  • Fallback: https://quantecon.github.io/QuantEcon.manual/pr-{number}/

Where {number} is your PR number (e.g., pr-45 for Pull Request #45).

Workflow Configuration#

The CI workflow is defined in .github/workflows/ci.yml and includes:

  • Environment Setup: Uses the same conda environment as production (environment.yml)

  • Build Process: Runs jb build manual --path-output ./

  • Safe Deployment: Uses subdirectory deployment to avoid conflicts with the main site

  • Smart Comments: Creates new preview comments for each build with commit SHA

  • Robust Cleanup: Removes preview directories when PRs are closed with:

    • Automatic detection of merged vs. closed PRs

    • Retry logic for push operations

    • Verification of successful cleanup

    • Graceful handling of edge cases

Note

The preview workflow is completely safe and won’t interfere with the main documentation site. It uses subdirectory deployment and preserves all existing content and configuration.

When the main branch publishes to gh-pages, the publish.yml workflow is configured with keep_files: true to preserve existing PR preview directories, so active previews remain accessible even after main branch deployments.

The cleanup mechanism has been enhanced to ensure previews are reliably removed when PRs are closed or merged, preventing accumulation of old preview directories.

GPU-Enabled Workflows#

For lecture repositories that require GPU computation, QuantEcon uses specialized workflows.

Our GPU based workflows are now built using RunsOn

Build Images#

There are two types of workflows:

  1. Using our own custom ami that is preloaded with CUDA, CUDANN, and LaTeX

  2. Using the ami provided by Runs On that includes Docker support. This is used by collab.yml workflows or any containerized workflow.

The benefit of using our own custom AMI is that we can choose CUDA, CUDANN versions and ensure we are able to use the latest versions to support lectures on JAX and GPU computing.

It is best to install only system wide software such as GPU frameworks, tool-kits and drivers.

When setting up a custom AMI we should install the NVIDIA support:

  1. nvidia CUDA framework

  2. nvidia drivers

  3. nvidia CUDANN

using the latest versions and then LaTeX

sudo apt-get -qq update
sudo apt-get install -y     \
texlive-latex-recommended \
texlive-latex-extra       \
texlive-fonts-recommended \
texlive-fonts-extra       \
texlive-xetex             \
latexmk                   \
xindy                     \
dvipng                    \
cm-super

Leave anaconda and local installs to the GitHub actions workflow.

While this incurs a small setup cost in terms of execution time, it provides flexibility in upgrading the python / anaconda software stack.

In addition to anaconda, you should install GPU-enabled Python packages in the GitHub Action as a setup step. The current standard across the GPU lecture repositories (e.g. lecture-jax, lecture-python-programming) is:

  - name: Install JAX and Numpyro
    shell: bash -l {0}
    run: |
        pip install -U "jax[cuda13]"
        pip install numpyro
        python scripts/test-jax-install.py

Note

The jax[cuda13] selector installs JAX against the CUDA 13 wheels and is matched to the CUDA / driver version baked into the custom AMI (see below). Bump this selector when the AMI is rebuilt against a newer CUDA release.

If a lecture also requires PyTorch / Pyro, add them as a separate install step:

  - name: Install PyTorch and Pyro
    shell: bash -l {0}
    run: |
        pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu130
        pip install pyro-ppl

Warning

PyTorch typically lags JAX in terms of compatibility with NVIDIA drivers. It is best to install the nightly versions to ensure the best compatibility, and to match the CUDA index URL (cu130, cu13x, …) to the CUDA version on the AMI.

You can check the latest on the PyTorch site with the installation table

Standard Lecture Workflows#

A typical QuantEcon lecture repository (e.g. lecture-python.myst) ships the following workflows in .github/workflows/:

Workflow

Trigger

Purpose

ci.yml

Pull requests

Build the lecture and generate a preview for review

cache.yml

Push to main (and schedule)

Execute the full project and save the _build execution cache as a workflow artifact

publish.yml

Push of a publish* tag

Build and deploy the live site (see Publishing Lectures)

linkcheck.yml

Schedule / manual

Validate internal and external links

collab.yml

Manual / containerized

Build inside a Docker container on a RunsOn runner (used for collab and container workflows)

PR and publish builds restore the _build artifact produced by cache.yml, so unchanged lectures are not re-executed. See Setup Cache for GitHub Actions for the configuration detail.

QuantEcon Actions — Reusable Build & Publish Library#

Note

This is the target architecture for QuantEcon lecture CI/CD. New repositories should adopt it and existing repositories are being migrated over time. See the lecture deployment table for the current per-repo status.

The QuantEcon/actions repository provides a set of reusable composite GitHub Actions that standardise and speed up lecture builds. Rather than each repository carrying a long, hand-maintained publish.yml, the build is composed from small, independently versioned actions (current release v0.6.0). lecture-dp is the reference implementation.

The design rests on three pillars:

  1. Containerised environment — a pre-built Docker image ghcr.io/quantecon/quantecon-build:latest (Ubuntu 24.04 + TexLive + Miniconda + Anaconda 2025.12 base + Jupyter Book tooling), rebuilt weekly and published to the GitHub Container Registry. This removes the 2–3 minute LaTeX install from every run. The container is built from the containers/ definitions in the same repo.

  2. Modular actions — small composable steps with clear responsibilities (see table below).

  3. Two-layer caching — the container image (environment) plus the Jupyter _build cache (execution), so only changed lectures are re-executed.

Available actions#

Action

Role

setup-environment

Flexible environment setup (optional Conda / LaTeX / ML libs); container-aware

build-lectures

Build Jupyter Book (HTML, PDF, notebooks) with unified error handling and asset assembly

build-jupyter-cache

Weekly execution-cache generation on main (validates all builders pass before saving)

restore-jupyter-cache

Read-only cache restore for PR builds (never writes, so PRs can’t corrupt the cache)

publish-gh-pages

Deploy to GitHub Pages using native artifact deployment (no gh-pages branch); optional custom domain and release assets

preview-netlify

Deploy PR previews to Netlify with smart PR comments

preview-cloudflare

Deploy PR previews to Cloudflare Pages (pr-N.project.pages.dev)

Important

publish-gh-pages deploys via the native GitHub Pages artifact pipeline (actions/deploy-pages), so repositories on this path do not have a gh-pages branch. The local ghp-import fallback documented in Building Locally and Push to GitHub Pages does not apply to them.

Minimal publish workflow#

A migrated publish.yml reduces to roughly:

jobs:
  build:
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/quantecon/quantecon-build:latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: quantecon/actions/restore-jupyter-cache@v0.6.0
      - uses: quantecon/actions/build-lectures@v0.6.0
        id: build
        with:
          builder: 'html'
  deploy:
    needs: build
    runs-on: ubuntu-latest
    permissions: { contents: write, pages: write, id-token: write }
    steps:
      - uses: quantecon/actions/publish-gh-pages@v0.6.0
        with:
          build-dir: _build/html
          github-token: ${{ secrets.GITHUB_TOKEN }}

See also

The migration guide and architecture overview in the quantecon/actions repository document the full set of inputs and the step-by-step process for converting a repository.

Repository Backup Workflow#

QuantEcon uses an automated backup system to protect repositories against data loss. The backup workflow is managed from the workflow-backups repository.

Overview#

The backup workflow:

  1. Runs weekly (Sunday at 2 AM UTC) via GitHub Actions

  2. Creates mirror backups of selected repositories (all branches, tags, and history)

  3. Stores backups securely in AWS S3

  4. Generates reports on backup status and storage usage

Repository Selection#

Repositories are selected for backup using pattern-based matching in config.yml:

backup:
  enabled: true
  organization: "QuantEcon"
  
  # Exact repository names
  repositories:
    - "QuantEcon.py"
    - "QuantEcon.jl"
  
  # Regex patterns for matching multiple repos
  patterns:
    - "lecture-.*"      # All lecture repositories
    - "quantecon-.*"    # All quantecon packages
  
  s3:
    bucket: "your-backup-bucket"
    prefix: "backups/"

Authentication Setup#

The backup workflow requires two types of authentication:

AWS Authentication (OIDC)#

AWS access uses OIDC authentication (no long-lived credentials):

  1. An IAM Identity Provider is configured for GitHub Actions

  2. An IAM Role with S3 permissions is created

  3. The role ARN is stored as a repository secret: AWS_ROLE_ARN

GitHub Token for Private Repositories#

Important

Fine-Grained Personal Access Token (PAT) Setup

For backing up private repositories, a Fine-Grained PAT must be created with READ-ONLY permissions:

  1. Go to GitHub Settings → Developer settings → Personal access tokens → Fine-grained tokens

  2. Click “Generate new token”

  3. Set the following:

    • Token name: quantecon-backup-readonly (or similar descriptive name)

    • Expiration: Set an appropriate expiration (recommend 90 days or custom)

    • Resource owner: Select the QuantEcon organization

    • Repository access: Select “All repositories” or specific repositories to backup

    • Permissions: Set Repository permissions to:

      • Contents: Read-only

      • Metadata: Read-only (automatically selected)

    • All other permissions should remain at “No access”

  4. Generate the token and store it as the REPO_BACKUP_TOKEN secret in the workflow-backups repository

Security Note: The token should have minimal READ-ONLY permissions. Never grant write access for backup operations.

For public repositories only, the default GITHUB_TOKEN works automatically with no additional setup.

Backup Storage Structure#

Backups are stored in S3 with the following structure:

s3://bucket-name/
├── lecture-python/
│   ├── lecture-python-20251127.tar.gz
│   ├── lecture-python-20251120.tar.gz
│   └── lecture-python-20251113.tar.gz
└── QuantEcon.py/
    └── QuantEcon.py-20251127.tar.gz

Each backup is a complete git mirror including all branches, tags, and full commit history.

Restoring from Backup#

To restore a repository from backup:

# 1. Download the backup from S3
aws s3 cp s3://bucket-name/repo-name/repo-name-20251127.tar.gz .

# 2. Extract the archive (contains a bare git mirror)
tar -xzf repo-name-20251127.tar.gz

# 3. Clone from the bare repo to create a working repository
git clone repo-name restored-repo

# 4. Verify the restoration
cd restored-repo
git branch -a   # View all branches
git tag         # View all tags

Manual Trigger#

The backup workflow can be triggered manually via GitHub Actions:

  1. Go to the workflow-backups Actions page

  2. Select the “Repository Backup” workflow

  3. Click “Run workflow”

More Information#

For detailed setup instructions and configuration options, see the workflow-backups README.