Run Sidekiq or Celery workers next to your web app

The moment an app needs to send an email, resize an image, or call a slow third-party API as a side effect of a request, that work needs to happen somewhere that isn’t the request itself. Sidekiq for Ruby and Celery for Python both solve the same problem: a queue holds the job, and a separate worker process pulls jobs off it and runs them, so a slow email provider never makes a page load feel slow.

This guide sets up a worker process next to an existing web app: Sidekiq with Redis as the queue backend for a Ruby app, or Celery with Redis as the broker for a Python app, both runnable with Docker Compose or as a plain systemd service.

What you’ll need

• A server already running your web app, or a fresh one. A Hetzner CX23 (2 vCPU, 4 GB RAM, 40 GB NVMe, €5.99/month excl. VAT) has enough headroom to run a web process, a worker process and Redis together for a small to medium app.
• Redis. If you don’t already have an instance, the official redis:8-alpine image in Compose is the fastest way to get one; Step 1 shows how.
• Your app’s code already structured so jobs are defined somewhere the worker process can import them: an app/jobs or app/sidekiq directory in Rails/Sidekiq terms, a tasks.py with @app.task decorators in Celery terms.

Step 1: Add Redis as the shared queue

Both Sidekiq and Celery need a broker to hold jobs between enqueue and execution. Redis is the common choice for both (Celery also supports RabbitMQ, but Redis covers the large majority of setups and keeps this guide to one moving part):

services:
  redis:
    image: redis:8-alpine
    volumes:
      - redis-data:/data
    restart: unless-stopped

volumes:
  redis-data:

If this Redis instance is purely a job queue and nothing else reads or writes to it, you don’t need the cache-tuning flags from a general-purpose Redis setup, default settings are fine; just make sure the volume is there so queued-but-not-yet-run jobs survive a restart rather than vanishing.

Step 2: Set up Sidekiq (Ruby)

Add the gem. This guide uses Sidekiq 8.1 (8.1.7 is the current release on RubyGems):

gem "sidekiq", "~> 8.1"

bundle install

Define a job:

class WelcomeEmailJob
  include Sidekiq::Job

  def perform(user_id)
    user = User.find(user_id)
    UserMailer.welcome(user).deliver_now
  end
end

Point Sidekiq at the Redis instance from Step 1 and set a sensible concurrency for the server’s size. Each Sidekiq thread holds its own database connection, so concurrency needs to match your database’s connection pool, not just how much CPU you have spare:

# config/sidekiq.yml
:concurrency: 5
:queues:
  - default
  - mailers

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.redis = { url: ENV.fetch("REDIS_URL") }
end

Sidekiq.configure_client do |config|
  config.redis = { url: ENV.fetch("REDIS_URL") }
end

Enqueue the job from anywhere in your app:

WelcomeEmailJob.perform_async(user.id)

Step 3: Set up Celery (Python)

Add the dependency. This guide uses Celery 5.6 (5.6.3 is the current release on PyPI); the redis extra installs the Redis client Celery needs:

pip install "celery[redis]==5.6.3"

Define the Celery app and a task in tasks.py, the layout Celery’s own first-steps guide uses:

import os
from celery import Celery

app = Celery("tasks", broker=os.environ["REDIS_URL"])

@app.task
def send_welcome_email(user_id):
    print(f"sending welcome email to user {user_id}")

Replace the body with your real work. Call the task from your web code with .delay(), which puts it on the queue and returns immediately:

from tasks import send_welcome_email

send_welcome_email.delay(user.id)

In a Django project, Celery’s “First steps with Django” guide puts the app in proj/celery.py and loads your Django settings first, so tasks can use the ORM; the worker command then becomes celery -A proj worker instead of celery -A tasks worker.

Step 4: Run the worker as its own process

Whichever library you’re using, the worker is a separate long-running process from your web server, not a thread inside it. With Docker Compose, that means a second service built from the same image as your web app, differing only in its command:

services:
  web:
    build: .
    command: bundle exec puma -C config/puma.rb
    env_file: .env
    ports:
      - "3000:3000"
    depends_on:
      - redis
    restart: unless-stopped

  worker:
    build: .
    command: bundle exec sidekiq -C config/sidekiq.yml -c 5
    env_file: .env
    stop_grace_period: 30s
    depends_on:
      - redis
    restart: unless-stopped

  redis:
    image: redis:8-alpine
    volumes:
      - redis-data:/data
    restart: unless-stopped

volumes:
  redis-data:

The Celery equivalent swaps the web and worker commands:

services:
  web:
    build: .
    command: gunicorn myproject.wsgi:application --bind 0.0.0.0:8000
    env_file: .env
    ports:
      - "8000:8000"
    depends_on:
      - redis
    restart: unless-stopped

  worker:
    build: .
    command: celery -A tasks worker --loglevel=INFO --concurrency=4
    env_file: .env
    depends_on:
      - redis
    restart: unless-stopped

  redis:
    image: redis:8-alpine
    volumes:
      - redis-data:/data
    restart: unless-stopped

volumes:
  redis-data:

stop_grace_period: 30s on the Sidekiq worker matters on every deploy. Docker sends TERM on stop and kills the container 10 seconds later by default, while Sidekiq waits up to 25 seconds by default for running jobs before it pushes unfinished ones back to Redis. Without the longer grace period, Docker kills Sidekiq before it can put those jobs back.

In both cases, the .env file next to the Compose file sets REDIS_URL=redis://redis:6379/0, so the web process and the worker point at the same Redis service by its Compose name.

Bring everything up:

docker compose up -d
docker compose ps

You should see the web, worker and redis services all with a status of Up.

Trigger a job from your app and confirm it ran by checking the worker’s logs:

docker compose logs -f worker

You should see the job name and a completion line (Sidekiq logs done: with a timing; Celery logs Task ... succeeded).

Step 5: The systemd alternative, no Docker

If your app runs directly on the host rather than in containers, a systemd unit gives you the same “runs alongside the web process, restarts on crash” behavior without Compose.

# /etc/systemd/system/myapp-sidekiq.service
[Unit]
Description=Sidekiq worker for myapp
After=network.target redis-server.service

[Service]
Type=notify
WatchdogSec=10
WorkingDirectory=/opt/myapp
EnvironmentFile=/opt/myapp/.env
ExecStart=/usr/local/bin/bundle exec sidekiq -C config/sidekiq.yml
User=deploy
Restart=always
RestartSec=1

[Install]
WantedBy=multi-user.target

Type=notify with WatchdogSec follows the example unit file in Sidekiq’s own repository, which works from Sidekiq 6.0.6 onward: Sidekiq tells systemd when it’s ready and keeps pinging it, and systemd restarts the process within seconds if those pings stop.

Celery doesn’t send those notifications, so it needs its own unit with Type=simple and no watchdog. A notify unit would sit waiting for a readiness signal that never comes, and systemd would eventually kill it:

# /etc/systemd/system/myapp-celery.service
[Unit]
Description=Celery worker for myapp
After=network.target redis-server.service

[Service]
Type=simple
WorkingDirectory=/opt/myapp
EnvironmentFile=/opt/myapp/.env
ExecStart=/opt/myapp/.venv/bin/celery -A tasks worker --loglevel=INFO --concurrency=4
User=deploy
Restart=always
RestartSec=1

[Install]
WantedBy=multi-user.target

Enable and start whichever one you use:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp-sidekiq
sudo systemctl status myapp-sidekiq

For Celery, use myapp-celery in the same three commands.

You should see active (running), and Restart=always means a crashed worker comes back on its own instead of silently staying down until someone notices jobs have stopped processing.

The shortcut: do it with Railyard

Everything in Steps 4 and 5 (a second process from the same image, its own restart policy, its own scaling) is what Railyard’s Workers tab sets up directly. It reads your repository for job runners it recognizes, Sidekiq, Solid Queue, GoodJob, Resque, Celery, RQ, Laravel’s queue worker or Horizon, an npm run worker script, or anything declared as a process in your Procfile, Procfile.dev or app.json, and offers each one as a worker you add in a click. Each worker runs your app’s own image and scales independently of your web process, so a traffic spike on the website doesn’t have to mean paying for more worker capacity too, or the reverse.

A worker-only app, one with no web process at all, can also be set to sleep when idle under Settings → General, and Railyard wakes it on its own schedule to drain anything queued before letting it go back to sleep, which matters if the queue is genuinely quiet most of the time and you’d rather not pay for a process sitting idle around the clock.

Troubleshooting

The worker starts but never picks up any jobs. Almost always a REDIS_URL mismatch between the process enqueuing jobs and the process running the worker, commonly because one reads it from an .env file and the other from a different one, or a stale value baked into an image at build time instead of read at runtime. Confirm both processes are pointed at the same Redis instance and the same logical database number, then check the queue has anything in it at all (redis-cli -u $REDIS_URL llen queue:default for Sidekiq’s default queue naming, or the equivalent for however your queue names are set up).

Jobs sit in the queue and never run, even though the worker is healthy. Check the queue names actually match. Sidekiq workers only pull from the queues listed in their own :queues config; a job enqueued to a queue the worker was never told to listen on sits there indefinitely without any error on either side. Celery has the same failure mode with its routing keys if you’ve configured custom queues rather than using the default.

Memory climbs until the server kills the worker. A worker with concurrency set higher than the server can actually support is the most common cause; each thread or process holds its own memory (and, for Sidekiq, its own database connection), so concurrency tuned for a bigger server than the one you’re actually running on will eventually get the process killed by the kernel’s out-of-memory handling. Lower concurrency first and watch whether the problem recurs before assuming it’s a leak in your job code.

The worker process exits immediately with no clear error. Usually a missing environment variable the worker needs but the web process doesn’t, since a worker’s code path through job classes often touches different parts of your app’s configuration. Run the exact worker command by hand in a shell with the same environment file to see the real exception instead of whatever your process manager summarized it as.

Two copies of a job somehow both ran. Neither tool promises exactly-once execution. A job that raises an error is retried, and with Celery’s acks_late setting turned on, a task interrupted mid-run is delivered again, so a job that had already done part of its work can repeat it. Write jobs so running them twice is harmless (check whether the email was already sent, use upserts instead of inserts), and that stops being a problem no matter how many workers you run.

FAQ

Should I use Sidekiq or Celery? Whichever matches your web framework’s language. There’s no meaningful reliability difference that should drive a Ruby app toward Celery or a Python app toward Sidekiq; both are mature, both default to Redis, and both support retries and scheduled jobs.

Does Celery need RabbitMQ instead of Redis? No. Celery supports both as a broker, and Redis is simpler to operate if you’re not already running RabbitMQ for something else. RabbitMQ has stronger delivery guarantees in some edge cases, which matters more at a scale most small apps aren’t at.

How many worker processes do I need? Start with one process at a modest concurrency (5 threads for Sidekiq, 4 for Celery, as shown above) and watch whether jobs back up in the queue under real load before adding more. A second worker process is simpler to reason about than raising one process’s concurrency past what the server’s CPU and your database’s connection pool can actually support.

What happens to a job if the worker crashes mid-run? It depends on how the worker stopped. On a normal shutdown (systemctl stop, docker compose stop, a deploy), Sidekiq waits for running jobs up to its shutdown timeout and pushes any unfinished ones back to Redis. If the process is killed outright (an out-of-memory kill, kill -9, a power loss), open-source Sidekiq loses the jobs it was running, because it removes a job from Redis when it starts working on it; Sidekiq Pro’s super_fetch exists to close that gap. Celery acknowledges a task before running it by default, so a hard crash loses that task too; setting task_acks_late = True makes Celery acknowledge after the task finishes, so an interrupted task is delivered again instead. If losing an occasional job matters, keep jobs idempotent and either turn on acks_late (Celery) or record important work somewhere durable before you enqueue it.

Do I need a separate Redis instance from my cache? Not necessarily, but if you do share one, keep the queue on its own logical database number or key prefix, and be careful with any maxmemory-policy that evicts keys under pressure, since a cache eviction policy applied to a queue can quietly drop a job that was never actually processed.

Once the queue and the worker are talking to the same Redis instance with matching queue names, the rest is mostly about sizing concurrency to what the server and database can actually support, not guessing at a bigger number because it sounds faster.