Replace Heroku review apps with pull request previews

Heroku’s review apps spin up a full disposable app for every pull request, which is genuinely useful for reviewers: click a link instead of reading a diff and imagining what it does. The part that’s easy to forget is that app.json requires addons and a region, every review app runs on Heroku’s normal dyno and add-on pricing, and an organization’s billing owner pays for every open PR’s app the same as a production one, pro-rated to the second but not free. This guide builds the same workflow yourself with GitHub Actions and Docker on a single server, then shows what a hosted version of this looks like once it accounts for the actual cost problem: idle preview apps.

By the end you’ll have a working per-PR preview system you built and understand completely, and know exactly what trade-off a hosted preview feature is actually making on your behalf.

What you’ll need

• A server your main app (or a staging copy of it) already deploys to as a Docker image. A Hetzner CX23 (2 vCPU, 4 GB RAM, €5.99/month excluding VAT) handles a handful of concurrent preview environments for a typical web app.
• A GitHub repository with Actions enabled, and a way to build and push a Docker image tagged per pull request (most CI setups already do this for the main branch; this guide tags by PR number instead).
• A wildcard DNS record or a subdomain per preview (this guide uses pr-<number>.review.example.com), and Caddy already running on the server for HTTPS.
• SSH access from GitHub Actions to the server (a deploy key is enough; this guide doesn’t cover setting that up, since most teams already have one for their main deploy).

Step 1: Build and tag an image per pull request

Whatever builds your production image already works here; the only change is the tag:

name: Build PR image
on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build and push
        run: |
          docker build -t ghcr.io/you/app:pr-${{ github.event.pull_request.number }} .
          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
          docker push ghcr.io/you/app:pr-${{ github.event.pull_request.number }}

Tagging by PR number rather than commit SHA means every push to the same PR overwrites the same tag, which is exactly the behavior a review app has: the preview tracks the branch, not one frozen commit.

Step 2: Write the per-PR Compose template

services:
  app:
    image: ghcr.io/you/app:pr-${PR_NUMBER}
    restart: unless-stopped
    env_file:
      - .env
    ports:
      - "127.0.0.1:${APP_PORT}:3000"

Binding to 127.0.0.1 keeps the container unreachable directly from the internet; Caddy is the only public entry point, matching how a single production server with multiple apps usually isolates them from each other on one host.

Step 3: Write the deploy script

This runs on the server itself, triggered over SSH from the Actions workflow:

#!/usr/bin/env bash
set -euo pipefail

PR_NUMBER="$1"
APP_PORT=$((40000 + PR_NUMBER))
DIR="/opt/previews/pr-$PR_NUMBER"

mkdir -p "$DIR"
cp /opt/previews/template/docker-compose.yml "$DIR/"
cp /opt/previews/template/.env "$DIR/.env"

export PR_NUMBER APP_PORT
docker compose -f "$DIR/docker-compose.yml" --project-directory "$DIR" pull
docker compose -f "$DIR/docker-compose.yml" --project-directory "$DIR" up -d

cat > "/etc/caddy/sites/pr-$PR_NUMBER.caddyfile" <<EOF
pr-$PR_NUMBER.review.example.com {
    reverse_proxy 127.0.0.1:$APP_PORT
}
EOF

docker exec caddy caddy reload --config /etc/caddy/Caddyfile

Deriving APP_PORT from the PR number (40000 + PR_NUMBER) is a simple way to avoid port collisions between simultaneously open PRs without a separate registry to track what’s taken; it works until you have more than a few thousand PRs open at once, which isn’t a real constraint for almost any team.

The main Caddyfile needs one line added once, up front, so it picks up every generated site file automatically:

import /etc/caddy/sites/*.caddyfile

Step 4: Wire it into the GitHub Actions workflow

  deploy-preview:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Deploy preview
        run: |
          ssh deploy@your-server "/opt/previews/deploy.sh ${{ github.event.pull_request.number }}"

Add a corresponding pull_request: types: [closed] job that runs a teardown script instead: docker compose down, remove the pr-<number> directory, delete the generated Caddyfile, and reload Caddy. A closed PR with a lingering container is the review-app equivalent of a server nobody remembered to turn off.

Step 5: Confirm it end to end

Open a test PR, wait for the workflow to finish, and visit https://pr-<number>.review.example.com; you should see that PR’s branch running, not the production app. Push a new commit to the same PR and confirm the image rebuilds and the same URL now serves the new commit. Close the PR and confirm the container and Caddy site file are both gone within a minute.

The shortcut: do it with Railyard

Everything above, the per-PR build, the dynamic port, the generated Caddy config, the teardown on close, is what Railyard’s preview environments feature does automatically once GitHub is connected and previews are turned on for an app. Every pull request gets its own full copy automatically, same build, same process types, its own URL, removed the moment the PR closes, with no workflow file or deploy script to maintain yourself.

The part worth calling out specifically, since it’s the actual problem with Heroku’s review apps rather than a minor detail: new preview and staging apps default to sleeping after 30 minutes of no traffic, waking on the next request. A PR open for two weeks while someone’s on vacation isn’t a container idling at full cost the whole time; it’s asleep almost all of it. That’s configurable per app under Settings → General (never, 10 minutes, 30 minutes, 1 hour, 6 hours, or a day of no traffic, or specific awake hours instead), and it’s the one thing the manual setup in Steps 1 through 5 doesn’t give you without building it yourself, since nothing in that script stops a container from running at full cost for as long as the PR stays open. There’s a per-app limit of 5 concurrent previews, which rarely matters for a typical team’s open PR count; Full reference: Environments and previews.

Troubleshooting

The preview URL 404s even though the deploy script ran. Check docker exec caddy caddy reload --config /etc/caddy/Caddyfile actually succeeded; a syntax error in a generated site file fails the reload silently unless you check its exit code, leaving Caddy running the previous configuration.

Two PRs serve the same content. APP_PORT collided, usually because the port derivation formula was changed without checking existing running previews first; docker ps --format '{{.Names}} {{.Ports}}' shows what’s actually bound where.

Containers pile up after PRs close. The teardown workflow either isn’t configured for the closed PR type, or GitHub didn’t fire it because the PR was closed without merging through a path your workflow’s types filter doesn’t cover; confirm types: [closed] catches both merged and simply closed PRs, since GitHub fires the same event for both.

The preview shows stale code after a push. Confirm the image tag in the Compose template actually matches what Step 1 just pushed; docker compose pull before up -d is required, since Compose won’t pull a new image for an existing tag unless told to.

FAQ

Does this work with GitLab instead of GitHub? The same pattern works with GitLab CI/CD instead of Actions; the per-PR Compose template, Caddy config generation and teardown script don’t change, only the workflow syntax that triggers them.

Can previews share a database with production? They can, but a migration running against a preview’s connection to the production database is a real risk worth avoiding; a per-preview database, even a cheap disposable one seeded from a snapshot, is safer and is what Railyard’s environments feature does by default for exactly this reason.

What happens to secrets each preview needs? The manual setup’s .env template is a real gap: it needs real values for anything the app requires (API keys, a database URL), and copying production secrets into every preview is its own security question worth thinking through separately, usually by scoping preview-only credentials from the start.

Is a CX23 really enough for several previews at once? For a handful of lightweight previews running the same app as one process each, yes; this is exactly the kind of workload a sleep-on-idle policy helps most with, since most open PRs aren’t being actively clicked through at any given moment.

Heroku’s review apps got one thing right that’s worth keeping: a reviewer should get a running link, not a diff to imagine. What it gets wrong is billing every open PR like a production server regardless of whether anyone’s actually looking at it, which Steps 1 through 5 above reproduce exactly unless you also build the sleep logic yourself. That’s the one piece worth being honest about before copying any preview-environment feature, hosted or self-built.