Notification System

Purpose: How to send notifications (SMS, push, WhatsApp, Slack) from any domain using the shared notification infrastructure.

Table of ContentsCopy Anchor Link

1. Overview
2. Architecture
3. Services Reference
4. Background Jobs
5. Notification Logging
6. Database Schema
7. Adding Notifications to a New Feature
8. Dev/Staging Behavior
9. Provider Details

OverviewCopy Anchor Link

The Shared::Notifications namespace provides four delivery services that any domain can use to send notifications. All services live in domains/shared/ and are domain-agnostic.

All SMS deliveries are automatically logged in shared.notification_logs for audit and duplicate detection.

ArchitectureCopy Anchor Link

Any Domain (clinical, billing, growth, ...)
    |
    |  Calls service directly or enqueues job
    v
+-----------------------------------------------+
|  Shared::Notifications                        |
|                                               |
|  SmsService -----> Twilio / MSG91             |
|  PushService ----> AWS SNS                    |
|  WhatsappService -> Legacy WhatsApp provider  |
|  SlackService ---> Slack Webhooks             |
+-----------------------------------------------+
    |
    v
shared.notification_logs  (audit trail)

File LocationsCopy Anchor Link

Services ReferenceCopy Anchor Link

Sending an SMS: a Sms::Base handlerCopy Anchor Link

Send every SMS through a handler under domains/shared/services/notifications/sms/ (#2709). Shared::Notifications::Sms::Base.deliver is the one send path. It resolves the message class from message_type, the language, the text (I18n, sms.* in config/locales/<code>.yml) and the DLT template id (Shared::MsgDetail.registered_dlt_id), then calls SmsService. A caller never passes text or a DLT id.

result = Shared::Notifications::Sms::SmsAppointment.deliver(
  to: '+919876543210',
  message_type: 'checked_in',
  params: { patient_first_name: 'Asha', clinic_name: 'Sunrise Clinic' },
  options: { tenant_id: tenant.id, locale: 'en' }, # :basic_language or :locale, :tenant_id, :sensitive
  recipient: patient,                               # gates on the recipient's SMS preferences
  facility: appointment.facility                    # gates on the facility's SMS settings
)

Gates (#2711)Copy Anchor Link

Each message class declares its gates with Shared::Notifications::Sms::Gating:

class FutureAppointmentReminder
  extend Gating
  gated_by event: 'appointment.reminder', category: 'appointments'
end

class Verification
  include CodeMessage
  extend Gating
  transactional!
end

• event is the SmsPolicy key. The facility's toggle for that event (EVENT_TOGGLES) and send_only_whatsapp apply. An event with no toggle is still stopped by send_only_whatsapp. Where the dispatcher sends the class, use the dispatcher's event key (gating_test.rb holds this, and fails on a declared event that is neither a dispatcher event, a toggled event nor listed as untoggled). The patient teleconsultation SMS have no toggle, so only send_only_whatsapp stops them.
• category is the ChannelPreferenceResolver category. The recipient's SMS switch for that category applies. With no category, only the recipient's SMS master switch applies.
• gated_by category: with no event skips the facility's settings, send_only_whatsapp included. The doctor's t-15 teleconsultation reminder is declared this way.
• transactional! skips both gates: codes, invites and account notices, practice suspended, the payment link, a failed payment, payment receipts, the consent notice, and the video call join link (Teleconsultation::VideoCallReady).

Base.deliver applies the gates when the caller passes recipient: and facility: as keywords. A domains/ caller that sends to a person passes both. A send the gates stop returns a failed, non-retryable SmsService::Result with error suppressed: <reason>, and logs one line ([Sms::Base] suppressed by SmsPolicy ... or [Sms::Base] suppressed by preference ...). An unknown message type still returns nil. A caller that must skip costly work (a join token, a short link) before the send asks Handler.suppressed_for?(message_type:, recipient:, facility:) first. Pass event: only when a class is sent for another purpose than its own, such as the queue text that AppointmentReminderJob sends as the 2-hour reminder. A call with no recipient:, facility: or event: never reaches the gating code. So the dispatcher, which applies both gates itself, and the legacy parts/ callers, which use the positional form, are not gated again. gating_test.rb fails on a message class with no declaration.

To add a message: add a message class with match? and get_message_text, declare its gates, register it in the handler's message_types, and put its text in en.yml under sms.<group>.<key>. The message class renders with Shared::Notifications::Sms::Text.t(language, :group, :key, **values). A message class holds no words of its own: text_in_i18n_test.rb fails on a string literal with a word in it. Code SMS go through Sms::SmsCode, which applies the code SMS cap and marks every send sensitive.

Language, DLT template and the guard (#2710)Copy Anchor Link

• Text = registered body. The text of a (message, language) is the body registered with DLT for that language, character for character. domains/lib/data/msg_templates.json stores each body and DLT category next to its id; script/build_msg_templates.rb takes them from the DLT export (lib/data/dlt_registered_templates.csv). template_conformity_test.rb renders every registered (message, language) and compares it with the body. Change the code to match the template, not the other way round. Take regional text from the export, never from a translation.
• Language. Base.deliver takes the requested language from options[:basic_language], else options[:locale], else LocaleResolver.for(recipient:, facility:) with the gate keywords, else en. It uses that language only when the message has both a text in that locale and a registered DLT id for it. Otherwise the text and the id are both English. They never disagree.
• Languages. Shared::Notifications::Languages.available is the one list: active reference.languages rows whose code has a locale file. en is always accepted, row or not. GET /reference/languages?purpose=notifications returns it; the CMS patient form and the patient app read it, and the patient, user and Identity::Facility#default_language fields validate against it. The CMS facility settings language is also the language of prescriptions and invoices (FacilitySettings#effective_language), so it lists and accepts every reference language. LocaleResolver.for picks the first of explicit, recipient language, facility language that is in the list, else en.
• Guard. An Indian number with no DLT id, or with a template variable longer than 30 counted characters (exempt special characters do not count, spaces do), is refused before MSG91 and logged as failed with the reason. A blank link or code (Text::REQUIRED_KEY) is refused for any number; another blank value, such as a facility with no public phone, is sent as it is. Free text (Text::CUT_KEYS: names, medicine, surgery, vaccine, camp and festival names, a review reply) is cut to 30 counted characters; links and codes are never cut. An SMS link is shortened with Shared::ShortenUrl.shorten(url, fallback: false), so a failure gives an empty link, not the long URL. A missing name is filled with Text.fallback(language, key): the word in the text's own language, never an English word in another language.
• Unregistered or unusable ids. A production id that DLT does not hold, or a registered body that cannot be sent (lib/data/dlt_unusable_templates.csv: broken placeholders, wrong script), is left out of the registry; that language falls back to English until it is registered again.

Async by default, synchronous for codes and invites (#2640)Copy Anchor Link

deliver claims the send (SmsService.claim: the checks, the duplicate window and a pending notification_logs row), then enqueues Shared::SendSmsJob with the row id and the tenant id, and returns a successful Result with queued: true. The number and the text stay in the row, out of Redis and Sidekiq Web. The job is enqueued only when the caller's transaction commits, so a write that rolls back leaves no row and sends nothing.

The send is synchronous, and the Result is the outcome of the send, in these cases:

Each claimed row carries an idempotency key: options[:idempotency_key], or a new sms:<uuid>. A caller that has a record for the send can pass its own key.

SmsServiceCopy Anchor Link

SmsService.call is internal. Only Sms::Base, Shared::SendSmsJob and the listed exceptions call it (the vendor practice sends and Growth campaigns, which have their own DLT id source for now); send_path_guard_test.rb fails on any other caller. It routes to the correct provider based on country code.

result = Shared::Notifications::SmsService.call(
  to: '+919876543210',
  message: 'Your appointment starts in 10 minutes',
  dlt_id: 'DLT_TEMPLATE_ID',  # Required for Indian numbers (regulatory)
  locale: 'en',                # Logged only; unicode comes from the text
  tenant_id: tenant.id
)

result.success?    # => true/false
result.log         # => Shared::NotificationLog record
result.error       # => error message on failure

Provider routing:

Built-in protections (#2640, authenticated sends, that is, sends with a tenant):

• Duplicate window: the same text to the same number within 10 minutes is not sent (Duplicate message). A code (sensitive: true) keeps a 10-second window. Only pending, sent and delivered rows count, so a failed send never blocks a resend.
• Idempotency key (idempotency_key:): the log row stores it. A later call with the same key does not send. It returns success with duplicate: true when the earlier attempt was sent, and a non-retryable failure when that attempt is pending or not confirmed.
• Not confirmed: a read timeout, an MSG91 5xx (a gateway can answer it after MSG91 took the text), or a 2xx with no readable body can mean the text is on the phone. The row is failed with metadata.unconfirmed = true, the result has retryable: false, and the key stays held. Only a connection that never opened (Net::OpenTimeout, refused, DNS) and an MSG91 408 or 429 set retryable: true, and the row then has metadata.retryable = true. Any other MSG91 4xx (a bad template, a bad number, an auth error) is permanent. Any other exception leaves retryable nil, and nothing retries it.
• NotificationLog.claim: the duplicate check and the insert of the pending row run under a session advisory lock on the number, released as soon as the row is written. It is never held during the provider call or until a caller's transaction ends, and a sender holds one number's lock at a time.
• The Slack mirror (staging) and the development outbox record only a send that went out, so a duplicate or a replay never looks sent.
• Dummy number filtering: numbers starting with a configured prefix are skipped
• Dev/staging: messages are routed to Slack instead of real SMS

PushServiceCopy Anchor Link

Send a push notification to a user's mobile devices.

Shared::Notifications::PushService.call(
  user: user,                    # User record (must have associated devices)
  title: 'Appointment Reminder',
  body: 'Your appointment starts in 10 minutes',
  key: 'appointment_reminder',  # Notification type key
  app_type: 'patient',          # patient | doctor | pharmacy | cms | clinic
  data: { appointment_id: '...' },  # Custom payload
  voice_call: false             # true adds 10s TTL on Android for urgent calls
)

How it works:

• Queries user.devices filtered by app_type
• Sends to each device via AWS SNS using the device's endpoint_arn
• If an Identity::Device has no endpoint_arn, schedules Shared::CreatePushEndpointJob. The job makes the SNS platform endpoint, saves its ARN, and enables a disabled endpoint of the same token again. The push goes at the next send. A legacy public.devices row still gets the parts job SnsCreatePlatformEndpoint.
• One device row owns a push token: the row that claimed it last (identity.devices.push_token_claimed_at). Each sign-in that sends the token claims it, and the other rows of that token lose their endpoints, so one user's notices do not reach a phone that another user signed in on later. Only a claim sets that column to a new time, so a trust, another save, or an endpoint that the job saves does not change the owner. Every sign-in path claims the token: password sign-in (SaveDeviceService), two-factor verify, PHR login, and signup verify (Identity::Device.find_or_create_by_device). A path that finds a device row calls Identity::Device#register_push_token!: the same token is claimed again, and a new token (a reinstalled app) is saved on the row, and the save claims it. When the unique index refuses the new token (the user has another row with it), the sign-in goes on and the fault logs [push] tag=token_claim_failed with the device id. The job asks for one endpoint for each device in ten minutes, drops a token that SNS refuses, and retries throttling and SNS server errors.
• A sign-in that claims a token and the job that saves the endpoint of that token take the same Postgres advisory locks (Identity::Device.lock_push_tokens, keyed by a digest of each token). The job checks the owner again under those locks after the SNS request. When another row claimed the token during the request, the job logs outcome=lost_race reason=not_owner and saves nothing. It keeps the SNS endpoint, because SNS gives the same endpoint to the same token, and the new owner uses it. A database fault in that write logs outcome=write_failed and the job runs again. A fault in a claim rolls back the claim only (a savepoint) and does not fail the sign-in. It logs [push] tag=token_claim_failed and reports Identity::Device::TokenClaimFailed as an error: until the next claim, the previous owner keeps the endpoint.
• A logout stops the push of its device. Identity::Device.sign_out_push runs when a session ends: logout (DELETE /identity/sessions/logout), the revoke of one session (the session list and the platform admin), and the revoke of all other sessions ("log out all other devices", a password change, a single-session sign-in). Logout ends the device that the details name, or, when no row matches, the device of the session of the request; it also signs out every other row of the user with the sent token. A device with no active, unexpired session left loses endpoint_arn and voip_endpoint_arn, and its push_token_claimed_at goes to PUSH_SIGNED_OUT_AT (1970-01-01, a time before every claim). The older rows of the same token (the users who signed in on the phone before) are signed out too, so none of them becomes the owner again. A signed-out row is never the owner. PushService skips that device ([push] tag=signed_out_skip) and asks for no endpoint. The job makes none (outcome=signed_out, or lost_race reason=signed_out when the logout came during the SNS request). The next sign-in on the device claims the token again, and the push starts again. The row and its token stay, so the trusted-device check still finds the device. A device with another active session keeps its push. A session whose device row is not found logs [push] tag=logout_no_device or tag=sign_out_no_device with ids only. A fault logs [push] tag=sign_out_failed and reports Identity::Device::PushSignOutFailed; the logout still succeeds.
• If no endpoint can be made (no platform application for the device type and app type, or no SNS client), each push writes [push] tag=no_endpoint reason=..., and the error tracker gets PushService::NoEndpoint at most once an hour for each process.
• Formats messages differently for Android (GCM) vs iOS (APNS)
• Sound: notify.mp3 (Android), notify.wav (iOS)

Disabled push endpointsCopy Anchor Link

When SNS disables an endpoint (EndpointDisabled), PushService clears the endpoint of the device and marks the device token in the cache. A marked token gets no push for 7 days (DISABLED_TOKEN_TTL). Each skipped push is counted, and the log line [push] tag=push_skipped_disabled_token count=N comes at most once an hour for each process.

The mark goes away when:

• the app registers its token again at sign-in (the same token or a new one);
• the device gets a new endpoint (endpoint_arn);
• 7 days pass.

Runbook: a bad APNs or FCM credential can disable every endpoint at once. The error tracker then shows PushService::EndpointsDisabled with a high disabled_endpoints count. Its context gives mark_version, mark_ttl_days and this page. After you fix the credential, clear every mark:

1. Set PUSH_DISABLED_MARK_VERSION to a value that is not the current mark_version (for example 2).
2. Restart the app and the workers.

To make the change permanent, increase DISABLED_MARK_VERSION in push_service.rb, and remove the variable.

Activity tracking (optional):

Shared::Notifications::PushService.call(
  user: user,
  title: 'New message',
  key: 'new_message',
  app_type: 'patient',
  activity: {
    object: message,           # Must respond to create_activity
    owner: sender,
    recipient: user,
    parameters: { extra: 'data' }
  }
)

SlackServiceCopy Anchor Link

Post a message to an internal Slack channel via webhook.

Shared::Notifications::SlackService.call(
  channel: :telemedicine,       # :telemedicine | :support | :bots | :billing
  title: 'New Appointment',
  color: '#7CD197',             # Hex color for the attachment bar
  fields: [
    { title: 'Doctor', value: 'Dr. Smith', short: true },
    { title: 'Patient', value: 'John Doe', short: true },
    { title: 'Notes', value: 'Follow-up visit', short: false }
  ]
)

Channels:

Webhook URLs are stored in Rails credentials under slack_kulcare_* keys.

WhatsappServiceCopy Anchor Link

Send a WhatsApp message using a pre-defined template. It takes idempotency_key: and applies the same claim, 10-minute duplicate window and read-timeout rule as SmsService (#2640). A message with a code parameter (4 to 8 digits) is not held to the window.

Shared::Notifications::WhatsappService.call(
  to: '+919876543210',
  template: 'telemed10',
  params: { patient_name: 'John', doctor_name: 'Dr. Smith', link: url }
)

Note: Currently delegates to the legacy WhatsApp::TelemedicineReminder provider. A full port is planned for a future phase. If the legacy provider is unavailable, the call is silently skipped with a warning log.

Background JobsCopy Anchor Link

Use these jobs when you want to send notifications asynchronously (recommended for most cases to avoid blocking requests).

SendSmsJobCopy Anchor Link

Sms::Base.deliver enqueues it (#2640). Do not enqueue it with a message: yourself: send_path_guard_test.rb fails on any caller outside Sms::Base, because a text built outside a message class has no DLT id.

• Arguments: log_id: and tenant_id: only. It reads the number, the text, the DLT id and the language from the row.
• Sets the RLS context from tenant_id, for the session and not in a transaction, so its row updates commit as they happen. The Sidekiq middleware captures the tenant at enqueue, which is after the caller's transaction has ended.
• Sends a row only when it is pending with no attempt started, or failed with metadata.retryable. It marks metadata.attempt_started_at before the provider call. A pending row with that mark (a worker stopped during the send) is ended as not confirmed and not sent again. A sent row, a permanently failed row and a code row are not sent.
• Raises RetryableFailure only when SmsService marks the failure transient (retryable: true), so ActiveJob retries the same row up to 5 attempts. Any other failure is not retried and is logged as [SendSmsJob] tag=not_sent or tag=duplicate. After the last attempt it logs [SendSmsJob] tag=retries_exhausted and stops; Sidekiq does not retry it.

Queue: sms_messages

SendPushJobCopy Anchor Link

Shared::SendPushJob.perform_later(
  user_id: user.id,            # Pass user_id, not user object
  title: 'Appointment Reminder',
  body: 'Your appointment starts in 10 minutes',
  key: 'appointment_reminder',
  app_type: 'patient',
  data: { appointment_id: '...' },
  voice_call: false
)

Queue: push_notifications

Note: The job looks up the user by ID from Identity::User first, falling back to User (legacy model).

Notification LoggingCopy Anchor Link

All SMS deliveries are automatically logged in shared.notification_logs via Shared::NotificationLog.

# Query recent failures
Shared::NotificationLog.by_channel('sms').failed.recent

# Check delivery history for a recipient
Shared::NotificationLog.where(recipient: '+919876543210').order(created_at: :desc)

# Count today's SMS
Shared::NotificationLog.by_channel('sms').where('created_at > ?', Time.current.beginning_of_day).count

Model DetailsCopy Anchor Link

File: domains/shared/models/notification_log.rb

Database SchemaCopy Anchor Link

shared.notification_logsCopy Anchor Link

CREATE TABLE shared.notification_logs (
  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  tenant_id       UUID,
  channel         VARCHAR NOT NULL,
  status          VARCHAR NOT NULL DEFAULT 'pending',
  recipient       VARCHAR NOT NULL,
  content         TEXT,
  template_id     VARCHAR,
  provider        VARCHAR,
  provider_id     VARCHAR,
  failure_reason  VARCHAR,
  metadata        JSONB DEFAULT '{}',
  created_at      TIMESTAMP NOT NULL,
  updated_at      TIMESTAMP NOT NULL,
  event           VARCHAR,
  read_at         TIMESTAMP,
  actor_id        UUID,
  idempotency_key VARCHAR                 -- #2640
);

CREATE INDEX idx_notif_logs_tenant ON shared.notification_logs (tenant_id);
CREATE INDEX idx_notif_logs_channel ON shared.notification_logs (channel);
CREATE INDEX idx_notif_logs_dedup ON shared.notification_logs (channel, recipient, created_at);
CREATE INDEX idx_notif_logs_created_at ON shared.notification_logs (created_at);
-- #2640: one claim for each key. A failed attempt does not hold it.
CREATE UNIQUE INDEX idx_notif_logs_idempotency_key
  ON shared.notification_logs (tenant_id, channel, idempotency_key)
  WHERE idempotency_key IS NOT NULL AND status IN ('pending', 'sent', 'delivered');

Adding Notifications to a New FeatureCopy Anchor Link

Here's the pattern for adding notifications to a new domain feature:

Step 1: Choose sync vs asyncCopy Anchor Link

Step 2: Send the notificationCopy Anchor Link

# Example: Billing domain sends payment confirmation
class Billing::PaymentConfirmationService
  def call(payment)
    # ... process payment ...

    # Send SMS confirmation (async: Sms::Base enqueues SendSmsJob)
    Shared::Notifications::Sms::SmsRequestPayments.deliver(
      to: payment.patient_phone,
      message_type: 'patient_payment_complete',
      params: { amount: payment.amount, doctor_name: payment.doctor_name },
      options: { tenant_id: payment.tenant_id, idempotency_key: "payment:#{payment.id}:received" },
      recipient: payment.patient
    )

    # Send push notification (async)
    Shared::SendPushJob.perform_later(
      user_id: payment.patient_user_id,
      title: 'Payment Confirmed',
      body: "Your payment of #{payment.amount} has been received",
      key: 'payment_confirmed',
      app_type: 'patient',
      data: { payment_id: payment.id }
    )

    # Alert team on Slack (sync, lightweight)
    Shared::Notifications::SlackService.call(
      channel: :billing,
      title: 'Payment Received',
      fields: [
        { title: 'Amount', value: payment.amount.to_s, short: true },
        { title: 'Patient', value: payment.patient_name, short: true }
      ]
    )
  end
end

Step 3: Create domain-specific notification jobs (optional)Copy Anchor Link

For complex notification logic (scheduling, reminder chains, conditional delivery), create domain-specific jobs that orchestrate the shared services:

# Example: domains/billing/jobs/payment_reminder_job.rb
class Billing::PaymentReminderJob < Shared::ApplicationJob
  queue_as :sms_messages

  def perform(invoice_id)
    invoice = Billing::Invoice.find_by(id: invoice_id)
    return if invoice.nil? || invoice.paid?

    # A handler and message class of its own, text in en.yml under sms.billing.*
    Shared::Notifications::Sms::SmsBilling.deliver(
      to: invoice.patient_phone,
      message_type: 'payment_reminder',
      params: { invoice_number: invoice.number, total: invoice.total },
      options: { tenant_id: invoice.tenant_id }
    )
  end
end

Key RulesCopy Anchor Link

1. Never call external APIs (SMS, push) inside a DB transaction -- use perform_later or call after the transaction commits
2. Send SMS through a Sms::Base handler, never SmsService.call with free text; use SendPushJob for simple fire-and-forget push
3. Create domain-specific jobs when you need guard clauses, scheduling, or multi-step notification logic
4. Always pass tenant_id in the handler options for the audit trail
5. For Indian numbers: the handler looks up the DLT id of the message class; register the template, add its id to lib/data/new_dlt_registrations.csv, add the template to lib/data/dlt_registered_templates.csv (the DLT export), and run ruby script/build_msg_templates.rb. Never hand-edit msg_templates.json

Dev/Staging BehaviorCopy Anchor Link

Provider DetailsCopy Anchor Link

SMS ProvidersCopy Anchor Link

Twilio (US/UAE):

• Credentials: us_sender_phone_number in Rails credentials
• Formats numbers with + prefix automatically

MSG91 (India):

• API: https://api.msg91.com/api/v2/sendsms
• Credentials: msg91_auth_key in Rails credentials
• Sender ID: KLCARE
• DLT template ID required for regulatory compliance
• Unicode support for non-English messages (set locale param)
• Non-retryable error codes: 301, 202, 311

Push ProviderCopy Anchor Link

AWS SNS:

• Device endpoint ARNs managed per device
• Platform-specific message formatting (GCM/APNS)
• Voice call TTL: 10 seconds on Android for time-sensitive notifications

SlackCopy Anchor Link

• Webhook URLs stored in Rails credentials: slack_kulcare_telemedicine, slack_kulcare_support, slack_kulcare_bots, slack_kulcare_billing
• Read timeout: 10 seconds

See AlsoCopy Anchor Link

• Teleconsultation -- example of a feature using all four notification channels with scheduled reminders
• Teleconsultation Testing Guide -- includes testing patterns for notification services