(tools:playwright)=
# Visual regression testing (Playwright)

QuantEcon's two theme repositories use [Playwright](https://playwright.dev) for **visual-regression
testing**: they render a small fixture site, take screenshots, and compare them pixel-for-pixel against
committed baseline images. A failing test means the theme changed how *something* looks — which is exactly
what you want to catch (and review) when working on a theme.

| Repository | What it themes | Playwright role |
|---|---|---|
| [quantecon-book-theme](https://github.com/QuantEcon/quantecon-book-theme) | The **Sphinx / Jupyter Book** theme (static HTML) | Snapshot every theme surface against a curated fixtures site |
| [quantecon-theme-src](https://github.com/QuantEcon/quantecon-theme-src) | The **MyST** runtime theme (Remix server) | Snapshot the rendered fixture + a WebKit FOUC guard |

```{note}
Visual tests catch what unit tests cannot: a CSS change that shifts spacing, a markup change that breaks an
admonition, or a dependency bump that re-renders math. Because the output is an image, "did this look
right?" is answered by a diff a human can review, not by an assertion someone had to write in advance.
```

## How it works (both repos)

The shared pattern is:

1. **Serve a fixture site** locally on a fixed port (Playwright's `webServer`).
2. **Visit each page** and call `expect(page).toHaveScreenshot(...)`.
3. Playwright compares the screenshot to the committed baseline under `tests/visual/__snapshots__/` and
   fails on a diff larger than the configured tolerance.
4. On an **intentional** change you regenerate the baselines (`--update-snapshots`) and commit the new
   images.

Both repos run **desktop Chrome (1280-wide)** and **mobile (Pixel 5)** viewports, store the test code in
`tests/visual/`, and keep baselines in `tests/visual/__snapshots__/<project>/`. Where they differ is in
*how the fixture is served*, because the two themes are fundamentally different artifacts.

### `quantecon-book-theme` — static HTML fixtures

The Sphinx theme produces static HTML, so the fixture is a pre-built site served with a plain HTTP server:

- The fixture site lives in a **separate repo**,
  [quantecon-book-theme-fixtures](https://github.com/QuantEcon/quantecon-book-theme-fixtures): a landing
  page, ~12 synthetic pages (one per theme surface — typography, code blocks, math, admonitions, exercises,
  proofs, tables, figures, …) plus real-world lecture captures that previously exposed theme bugs.
- It is **pinned to a specific commit** (`FIXTURES_SHA` in the workflows) so the input doesn't move between
  theme PRs.
- `playwright.config.ts` serves it with `python -m http.server` and points `baseURL` at it.
- `theme.spec.ts` loops over the fixture pages taking a full-page screenshot plus header (`.qe-page__header`)
  and sidebar (`.qe-sidebar`) region snapshots, and adds targeted tests for dark mode, code blocks,
  f-string token styling, MathJax, typography, and definition lists.

```{note}
Because the fixtures site is rendered ahead of time (notebooks are **not** executed), every page is
deterministic — there is no flaky plot output to skip. Pages with rendered **math** use a looser tolerance
(`maxDiffPixelRatio: 0.05`, vs. the strict `0.01` default) because MathJax font metrics shift slightly
between Ubuntu CI and macOS.
```

### `quantecon-theme-src` — runtime MyST server

The MyST theme is consumed by lectures as a built bundle served by a live Remix/`myst` server, so the
fixture is rendered at run time:

- The fixture is a minimal MyST project in `tests/visual/fixture/` (`intro.md`, `features.md`,
  `notebook.ipynb`).
- `serve.sh` injects the **theme under test** into the fixture's `myst.yml` via the `THEME_TEMPLATE`
  environment variable, then runs `myst start`. `THEME_TEMPLATE` accepts either a **local theme build
  directory** (the candidate) or a **GitHub archive zip URL** (a deployed/released bundle) — so you can diff
  any two theme versions against identical content.
- `theme.spec.ts` takes one full-page snapshot per surface; the `notebook` page matters most because it
  exercises the `@myst-theme` output-node rendering.

(tools:playwright:fouc)=
#### The FOUC guard (WebKit-only)

`quantecon-theme-src` also ships `fouc.spec.ts`, a **flash-of-unstyled-content** guard for Safari/WebKit
([#66](https://github.com/QuantEcon/quantecon-theme-src/issues/66)). It aborts all external stylesheets so
only the inline critical CSS in `app/root.tsx` can reach the first paint, then asserts the layout and font
are already correct. It is **snapshot-free** (it checks computed `display` / `font-family`, not pixels), so
it is robust across `myst` and CI versions. It runs on the `webkit-fouc` project only — Chromium paint-holds
and cannot exhibit the flash.

## Running the tests locally

`````{tab-set}
````{tab-item} quantecon-book-theme
The `tox` environments handle everything, including cloning the fixtures repo into `./fixtures/`:

```bash
# Run visual tests (clones fixtures if not present, builds them, runs Playwright)
tox -e visual

# Update baselines after an intentional change
tox -e visual-update

# Pin to a specific fixtures commit for local testing
FIXTURES_REF=<sha> tox -e visual
```

Prefer the manual route? `tests/visual/README.md` documents the by-hand setup (`git clone` the fixtures,
`pip install -e .`, `jb build`, `npm install`, `npx playwright install chromium`, `npm run test:visual`).
````

````{tab-item} quantecon-theme-src
Build the candidate theme, then point `THEME_TEMPLATE` at it:

```bash
# Build the candidate theme from this branch into .deploy/quantecon-theme
make build-theme

# Diff it against the committed baselines
THEME_TEMPLATE="$PWD/.deploy/quantecon-theme" npm run test:visual

# Refresh baselines once the diffs are confirmed intentional
THEME_TEMPLATE="$PWD/.deploy/quantecon-theme" npm run test:visual:update

# Run just the WebKit FOUC guard
THEME_TEMPLATE="$PWD/.deploy/quantecon-theme" npm run test:fouc
```

Requires the `mystmd` CLI (`myst`) on `PATH` and the browser binaries
(`npx playwright install --with-deps chromium webkit`).
````
`````

```{warning}
Snapshots are **platform-specific** — font rendering differs between macOS and Linux. In
`quantecon-book-theme`, the committed `tests/visual/__snapshots__/` baselines are the **Ubuntu/CI** images;
local macOS runs use a separate, gitignored `tests/visual/macos/` directory (selected via `SNAPSHOT_DIR`).
This means **the authoritative baselines are regenerated in CI**, not from your laptop — see below.
```

## Updating baselines

### `quantecon-book-theme` — via PR slash commands

Because the committed baselines are the Ubuntu/CI images, you regenerate them **on the PR**, not locally.
Comment on the pull request:

| Comment | Effect |
|---|---|
| `/update-snapshots` | Regenerates **all** baselines with `--update-snapshots`, uploads a before/after diff artifact, and commits the new images to your PR branch. Use when the theme styling legitimately changed. |
| `/update-new-snapshots` | Generates only **missing** baselines — existing images are left untouched. Use when you've added a new test/fixture page. |

The `.github/workflows/update-snapshots.yml` workflow performs the regeneration and pushes the commit.

```{tip}
First-time CI setup for a new branch: CI will fail because no Ubuntu baselines exist yet. Comment
`/update-new-snapshots` to seed them, then push your real changes.
```

### `quantecon-theme-src` — locally

`quantecon-theme-src` regenerates baselines **locally** with `npm run test:visual:update` (see the tab
above) and you commit the resulting images. Its CI runs only the **FOUC guard**, not the full snapshot
suite — the pixel snapshots are a local/manual validation step, so review `playwright-report/` before
committing updated baselines.

## CI integration

`````{tab-set}
````{tab-item} quantecon-book-theme
`.github/workflows/ci.yml` splits into three jobs so the preview deploys **even when visual tests fail** —
reviewers need the preview to decide whether a failure is an intentional design change or a regression:

```
build  →  visual   (gate; may fail)
       ↘
         preview   (always runs; Netlify deploy reviewers eyeball)
```

`build` renders the pinned fixtures site once; `visual` and `preview` consume it in parallel. The `visual`
job posts a "🎭 Visual Regression Test Results" summary comment and uploads the `playwright-report` artifact
(open its `index.html` for diffs). Separately, `tests.yml` runs the `pre-commit` and `pytest` gates.
````

````{tab-item} quantecon-theme-src
`.github/workflows/ci.yml` runs **Build & Typecheck** (`npm run compile`, `npm run prod:build`) and the
**FOUC guard (WebKit)** job, which builds the candidate theme with `make build-theme` and runs
`npm run test:fouc`. The full Chromium snapshot suite is **not** run in CI; it is the local validation step
described above.
````
`````

## Bumping the fixtures pin (`quantecon-book-theme`)

The fixtures repo is pinned by `FIXTURES_SHA`, which appears in **two** workflow files —
`.github/workflows/ci.yml` and `.github/workflows/update-snapshots.yml`. To adopt a new fixtures commit
(e.g. after adding a fixture page upstream):

1. Merge the new page in
   [quantecon-book-theme-fixtures](https://github.com/QuantEcon/quantecon-book-theme-fixtures).
2. Open a PR on `quantecon-book-theme` updating `FIXTURES_SHA` in **both** files (and add a matching test in
   `theme.spec.ts` if the new page needs one).
3. Comment `/update-new-snapshots` on that PR to seed baselines for the new page.

## When you change a theme — checklist

1. Make your change and run the visual tests locally (`tox -e visual`, or `make build-theme` +
   `npm run test:visual`).
2. If pixels changed, decide whether the change is **intentional**.
   - **Intentional** → regenerate baselines (`/update-snapshots` on the PR for `quantecon-book-theme`;
     `npm run test:visual:update` for `quantecon-theme-src`) and commit them.
   - **Unintentional** → it's a regression; fix it.
3. Review the diff (the `playwright-report` artifact / Netlify preview) and reference it in the PR so a
   reviewer can confirm the visual change is what you intended.

```{seealso}
The canonical, repo-local details live alongside the tests:
[`quantecon-book-theme/tests/visual/README.md`](https://github.com/QuantEcon/quantecon-book-theme/blob/main/tests/visual/README.md)
and
[`quantecon-theme-src/tests/visual/README.md`](https://github.com/QuantEcon/quantecon-theme-src/blob/main/tests/visual/README.md).
This page is the cross-repo overview; those READMEs are the source of truth for each setup.
```