Canary deploys that actually watch error rate, not just a timer

A standard deploy is binary: the new release passes its health check, and all traffic moves to it at once. That’s fine until the thing that breaks isn’t a crash on boot but a slow leak that only shows up under 100% of production load — by the time it’s obvious, everyone already has the bad version. A canary deploy exists specifically for that gap.

What actually happens

Turn it on under Settings → General → Deploy strategy, choose Canary, and set a canary share (1-99%): that percentage of new visitors gets the new release, running right next to the current one, not replacing it yet. A cookie keeps each visitor on whichever version they landed on first, for a day — nobody gets bounced between old and new mid-session.

From there, three settings decide how cautious the rollout is:

• Ramp steps, an ascending list like 5, 25, 50 — each step is its own canary, watched the same way, and clearing the last one promotes to 100%. Leave it blank for a single step you promote by hand.
• Dwell per step, in minutes — leave it blank and you promote each step yourself; set it and Railyard advances automatically once a step has run that long without tripping the next setting.
• Roll back above a 5xx rate (%) — the part that makes this more than a traffic-splitting toggle. If a step’s error rate crosses this threshold, Railyard rolls it back on its own. Nobody has to be watching a dashboard at 2am for this to catch a bad release.

While it’s live, the deploy page shows the canary’s request count, 5xx rate and p95 latency next to the rest of the app’s — the actual comparison that decides whether to promote, not a gut feeling.

What still has to run as a standard deploy

A few cases always skip canary entirely, and knowing them up front avoids confusion later: an app’s very first deploy, a deploy during maintenance mode, a worker-only app (nothing to split traffic to), and an app with a data volume. The release command — migrations, typically — still runs once, before the canary starts, which means migrations have to work against both the old and new code at the same time for however long the canary runs. That’s not a Railyard-specific constraint; it’s the actual tradeoff every canary-deploy system makes, stated plainly instead of glossed over.

Combining it with a deploy window

Canary answers “how fast does a new release reach everyone.” A separate, composable setting — deploy windows — answers “when are deploys allowed to happen at all”:

railyard deploy-window my-app mon,tue,wed,thu,fri 09:00 17:00 America/Toronto

Pushes outside the window queue and go out together when it opens; a manual deploy outside it asks first. The two features don’t interact beyond that — a canary that’s mid-ramp when a window closes keeps running exactly as configured, since ramping an already-started canary isn’t the same action as starting a new deploy.

Why this matters more once you own the servers

A canary with automatic rollback is one of the places where “self-hosted” could plausibly mean “fewer guardrails than a managed platform” — and this is the proof it doesn’t have to. The mechanism is the same regardless of who owns the box: watch real error rate against a real threshold, not a fixed timer, and let the system protect itself before a human has to.

Full reference, including scheduled deploys and deploy notes: Canary deploys and deploy windows.