Background Jobs Configuration

This document covers how background jobs actually run in the Kulcare API backend: Sidekiq (via systemd, managed by Capistrano), scheduled jobs (via whenever crontab, not Sidekiq-cron), job code layout (domains/*/jobs/), and failure alerting (Slack — there is no Sentry in this app).

Companion reading

• typesense-search-sync.md — the most detailed worked example of a job pipeline (§7 failure handling, §8 observability, "Deploying & operating").
• row-level-security.md — why jobs must run under the right tenant GUC (§5.5, cross-tenant fan-out).

The model in one paragraphCopy Anchor Link

Jobs are Active Job classes (queue_adapter = :sidekiq, set per environment in config/environments/{development,staging,production}.rb) that live under domains/<domain>/jobs/ and inherit from Shared::ApplicationJob. One Sidekiq process per server works every queue listed in config/sidekiq.yml — it's a systemd unit (sidekiq) managed by capistrano-sidekiq, so cap deploy restarts it automatically. Recurring work is cron, generated by the whenever gem from config/schedule.rb and applied automatically on every deploy (via Capfile → whenever/capistrano). Failures alert to Slack in production via a global Sidekiq error handler; everything else is visible in the Sidekiq Web UI at /sidekiq.

Sidekiq configurationCopy Anchor Link

config/sidekiq.yml (the real file)Copy Anchor Link

---
:verbose: false
:concurrency: 3
:pidfile: tmp/pids/sidekiq.pid
staging:
  :concurrency: 5
production:
  :concurrency: 5
:queues:
  - patient_history
  - knowledge_base
  # … ~55 more domain queues …
  - [mailers, 2]        # weighted 2x
  - reindexing
  - default
  - notifications
  # … etc …

Notes:

• Redis is not configured here. The connection comes from the initializer (below) via Rails credentials. Do not add a :redis: block to this file.
• Queues are bare names (reindexing, notifications, mailers, …). There is no queue_name_prefix — the prefix line in config/environments/production.rb is commented out.
• Adding a new queue? It must be listed here. Sidekiq only polls queues named in this file; a queue_as :clinical job with no entry below enqueues and then never runs (the #1855 QA bug). Whenever a domain adds a queue, add the entry in the same change.

config/initializers/sidekiq.rb (the real file, abridged)Copy Anchor Link

# Redis via Rails credentials (staging/production; dev/test fall back to
# Sidekiq's default localhost Redis):
unless Rails.env.development? || Rails.env.test?
  Sidekiq.configure_server do |config|
    config.redis = {
      url: Rails.application.credentials.redis_url,
      password: Rails.application.credentials.redis_password
    }
  end

  Sidekiq.configure_client do |config|
    config.redis = {
      url: Rails.application.credentials.redis_url,
      password: Rails.application.credentials.redis_password
    }
  end
end

# Global defaults: 3 retries, no backtrace (memory)
Sidekiq.default_job_options = { backtrace: false, retry: 3 }

Sidekiq.configure_server do |config|
  config.on(:startup) { Rails.logger.info 'Sidekiq is starting...' }
  config.on(:quiet)   { Rails.logger.warn 'Got USR1, stopping further job processing...' }
  config.on(:shutdown){ Rails.logger.warn 'Got TERM, shutting down process...' }

  # Failure alerting: Slack, production only
  config.error_handlers << proc { |ex| SlackSidekiqFailures.perform(ex) if Rails.env.production? }
end

# Sidekiq Web UI basic auth (credentials-stored)
Sidekiq::Web.use(Rack::Auth::Basic) do |username, password|
  username == Rails.application.credentials.sidekiq_username &&
    password == Rails.application.credentials.sidekiq_password
end

Tenant context in jobs (RLS)Copy Anchor Link

Many domain tables are RLS-forced, so jobs must execute under the right tenant GUC or they silently query nothing. This is handled by middleware — config/initializers/sidekiq_tenant_middleware.rb:

1. Client middleware captures Shared::Current.tenant / .user into the job payload at enqueue time.
2. Server middleware restores them (and the Postgres RLS GUC) on the worker, and clears everything in an ensure block to prevent context leakage between jobs.

Jobs therefore "just work" when enqueued from a tenant-scoped web request. For cron-enqueued or console-enqueued jobs (no ambient tenant), the job itself must establish context — see Identity::Typesense::BaseSyncJob#with_tenant_context for the canonical pattern and row-level-security.md §5.5 for the cross-tenant fan-out.

Job classes and organizationCopy Anchor Link

Jobs live under the domain that owns them — not app/jobs/:

backend/domains/
├── identity/jobs/            # e.g. device_prune_job.rb, typesense/*, geocode_facility_location_job.rb
├── billing/jobs/             # e.g. stale_pending_cleanup_job.rb, care_package_orders/*, marketplace/*
├── clinical/jobs/
├── growth/jobs/
├── notifications/jobs/
├── connect/jobs/
└── shared/jobs/application_job.rb

The base class (domains/shared/jobs/application_job.rb):

module Shared
  class ApplicationJob < ActiveJob::Base
    include Bullet::ActiveJob if defined?(Bullet::ActiveJob) && Rails.env.development?
  end
end

Conventions:

• Inherit from Shared::ApplicationJob (a legacy bare ApplicationJob exists in parts/core/jobs/ for older code; don't use it for new jobs).
• Declare the queue explicitly: queue_as :billing — and make sure that queue is in config/sidekiq.yml (see above).
• Retries are global (retry: 3, backtrace: false); a job that keeps failing exhausts retries and lands in the dead set, visible in the Web UI.
• Make cron-invoked jobs idempotent — cron and Sidekiq retries can double- fire (every entry in config/schedule.rb is written with this assumption).

Scheduled jobs (whenever → crontab)Copy Anchor Link

There is no sidekiq-cron, no config/schedule.yml, and no :scheduler: block in sidekiq.yml. Recurring work is plain cron, generated from backend/config/schedule.rb by the whenever gem. The Capfile wires it into Capistrano:

require 'whenever/capistrano'
set :whenever_identifier, ->{ "#{fetch(:application)}_#{fetch(:stage)}" }

so cap deploy regenerates the crontab automatically — there is no manual crontab step. The current schedule (all off-peak, all idempotent):

Cron output goes to /var/log/cron_log.log (set :output in schedule.rb).

Adding a scheduled job: add the entry to config/schedule.rb with a comment explaining why it's safe to re-run, then deploy — Capistrano applies it. Locally you can preview/apply manually:

bundle exec whenever                       # preview generated crontab
bundle exec whenever --update-crontab      # apply (deploy does this for you)
crontab -l                                 # verify what's installed

Deployment & the systemd unitCopy Anchor Link

capistrano-sidekiq (see Gemfile) provides a user-level systemd unit named sidekiq (install_plugin Capistrano::Sidekiq::Systemd in the Capfile), reading config/sidekiq.yml. Consequences:

• cap deploy runs DB migrations and then restarts the sidekiq unit — no manual service management during a normal deploy.
• The unit must exist on a new server before the first deploy (cap sidekiq:install / ops provisioning).
• One process works all queues from config/sidekiq.yml. If you ever split Sidekiq into per-queue processes, ensure every queue — especially :reindexing — is still covered by some worker, or that pipeline silently stops (the failure class #1592 was originally about).

Manual operations on a server (user systemd):

systemctl --user status sidekiq
systemctl --user restart sidekiq
journalctl --user -u sidekiq -f

Monitoring, failure handling & alertingCopy Anchor Link

How a failing job flows through the system:

1. The job raises → Sidekiq retries (global retry: 3).
2. Each failure fires the global error handler → SlackSidekiqFailures.perform(ex) — production only — which posts the error to the slack_be_issues webhook channel (parts/core/services/slack_sidekiq_failures.rb).
3. After retries are exhausted the job lands in the dead set, visible in the Sidekiq Web UI.

Observability surfaces:

• Sidekiq Web UI — mounted at /sidekiq (parts/core/routes.rb), basic- auth'd via credentials (initializer). Shows queue depths, retries, and the dead set in every environment. The sidekiq-failures gem adds a failures view.
• Slack alerts — production job failures (above).
• rake typesense:status — reindexing queue depth, retry/dead-set counts, collection document counts; usable during an incident.
• There is no Sentry. No sentry-* gem is installed; config/initializers/sentry.rb is fully commented out. (A queue named sentry appears in sidekiq.yml — it's a legacy queue name, not Sentry integration.) Don't add Sentry.* calls expecting them to fire; ops alerting is Slack.

Local developmentCopy Anchor Link

• config/environments/development.rb also sets config.active_job.queue_adapter = :sidekiq, so enqueueing in dev enqueues for real. Run a worker with bundle exec sidekiq -C config/sidekiq.yml (defaults to localhost Redis — no credentials needed), or use perform_now/inline adapters in tests.
• Dev/test skip the credentials-based Redis config (see the unless guard in the initializer).
• after_commit callbacks don't fire under transactional tests — see how the Typesense specs invoke callbacks directly (docs/guides/developer/typesense-search-sync.md §11).

TroubleshootingCopy Anchor Link

Job enqueued but never runs — the #1 cause: its queue is missing from config/sidekiq.yml (#1855). Add the queue, deploy (restarts the unit).

"Search is stale" / sync silently stopped — see typesense-search-sync.md §10: rake typesense:status → check the dead set → rake typesense:reindex.

A cron job didn't run — crontab -l (was the last deploy's whenever update applied?); check /var/log/cron_log.log; confirm the entry is in config/schedule.rb.

Job queries return nothing under RLS — the tenant GUC wasn't set: the job was enqueued without ambient context (cron/console). See the middleware notes above and row-level-security.md §6.1.

Sidekiq won't start — check Redis reachability with the credential URL, then journalctl --user -u sidekiq.

Next stepsCopy Anchor Link

• Monitoring
• Server Configuration
• Environment Configuration