How public search (patient-app doctor/facility discovery, Connect Discover) is
backed by Typesense, and how the index is kept in sync with Postgres. Read
this before touching anything that indexes to Typesense, adds a searchable
model, or changes a field that search reads.
Companion reading: row-level-security.md — the sync
jobs depend on the RLS patterns (especially §5.5, cross-tenant fan-out). The
indexed source tables are RLS-forced, so a job that re-fetches without the
right GUC silently indexes nothing.
Two Typesense collections back public search: practitioners_<env> (one
document per Identity::PractitionerRole, denormalizing the practitioner +
role + facility) and facilities_<env> (one document per Identity::Facility).
There is no vendor collection — vendors are not searchable in v1. The index
is kept fresh by direct Sidekiq jobs: a model after_commit hook enqueues a
small, model-specific job that re-fetches the record under the right tenant
context and upserts (or removes) its Typesense document. There is no buffer
table and no polling drain — the enqueue is the work (this replaced an earlier
Postgres-queue + scheduled-drain design; see §9). A nightly full reindex is the
self-healing backstop.
The whole thing leans on three invariants: the indexed tables are RLS-forced, so
every sync job must run under the record's tenant GUC (or it silently no-ops —
the #1489 trap); the index is eventually consistent (a brief staleness window is
acceptable, data is always derivable from Postgres via reindex); and the
read-side normalizer must tolerate two location shapes (a Typesense geopoint
array vs the Postgres-fallback hash — the #1596 trap).
The concern owns only the callback lifecycle + the swallow-rescue; each model
declares which job it enqueues via enqueue_typesense_sync(operation). The
jobs are deliberately model-specific (no generic dispatcher):
IndexPractitionerRoleJob(role_id, operation, tenant_id) — one
practitioners_<env> doc. upsert re-fetches the role and indexes it (or
removes the doc if the row is gone — the delete-between-enqueue-and-run race);
delete removes by id.
IndexFacilityJob(facility_id, operation, tenant_id) — one
facilities_<env> doc. Fires on any facility change (the facility doc
carries fields beyond the subset denormalized onto role docs).
ReindexFacilityRolesJob(facility_id, tenant_id) — re-indexes every
role doc at a facility. Fires only when a denormalized field changed (§4).
ReindexPractitionerJob(practitioner_id, operation) — cross-tenant
fan-out: re-indexes all of a practitioner's role docs across every tenant they
have a role in. Used by Practitioner and Connect::ProfessionalProfile
(both denormalize onto every role doc). No delete branch — there is no
standalone practitioner doc (see §6).
practitioners_<env> denormalizes facility fields (name, city, address, lat/lng,
location_phone) onto every role doc. So a facility/location change to one of
those must fan out to all roles — but an unrelated facility touch
(ai_settings_id, an updated_at-only touch, a bulk import) must not, or it
would enqueue thousands of spurious role reindexes.
Identity::FacilityLocation::DENORMALIZED_TO_FACILITY = address fields +
lat/lng/phone/is_primary/active — resync_parent_facility on
create/destroy and on the guarded update.
The facility's own doc (IndexFacilityJob, via TypesenseIndexable) refreshes
on any change; only the role cascade (ReindexFacilityRolesJob) is guarded.
If you add a field that search reads off a role doc, add it to
DENORMALIZED_TO_PRACTITIONERS (or the location list) or the change won't
propagate.
PractitionerRole and Facility are RLS-forced. A job that re-fetches them
without the matching app.current_tenant_id GUC active gets zero rows back and
silently no-ops (or, under the public_read carve-out, sees only active
rows). So:
Tenant-scoped jobs (IndexPractitionerRoleJob, IndexFacilityJob,
ReindexFacilityRolesJob) rely on the Sidekiq tenant middleware
(config/initializers/sidekiq_tenant_middleware.rb), which captures
Shared::Current.tenant at enqueue and restores the RLS context on the
worker. The tenant_id argument they accept is a defensive fallback for
context-less enqueues (cron/console/reconciler). BaseSyncJob#with_tenant_context
prefers the ambient context, falls back to tenant_id, and raises if
neither is present rather than running without a GUC.
ReindexPractitionerJob is the cross-tenant exception (the model has no
tenant_id). It iterates tenants and sets the GUC per tenant with
RlsContext.with(tenant.id) — the canonical fan-out from
row-level-security.md §5.5. Each tenant is isolated
with its own rescue so one tenant's failure doesn't starve the rest; it logs
per-tenant and re-raises an aggregate at the end (so Sidekiq still retries).
Cost note: ReindexPractitionerJob currently scans all tenants per
practitioner edit. Bounding it to the practitioner's actual tenants needs a
cross-tenant read of the RLS-forced practitioner_roles that doesn't depend
on the ambient GUC — deferred; acceptable at current tenant counts.
Single-doc deletes (a PractitionerRole destroy) enqueue
IndexPractitionerRoleJob(role_id, :delete), which removes the doc by id (no
re-fetch, no RLS needed).
Cascade deletes are handled by the association, not by capturing IDs:
Identity::Facility and Identity::Practitioner both
has_many :practitioner_roles, dependent: :destroy, and PractitionerRole
includes TypesenseIndexable — so destroying a facility or practitioner fires
each role's own on: :destroy callback → IndexPractitionerRoleJob(:delete).
The facility's own doc is removed by IndexFacilityJob(:delete); the practitioner
has no standalone doc, so ReindexPractitionerJob skips :delete entirely.
Enqueue failures (Redis unreachable when the after_commit tries to push):
swallowed by Shared::Typesense::EnqueueErrors.guard, which catches only
transport errors (RedisClient::Error, ConnectionPool::Error,
Redis::BaseError if loaded) and logs. A NotImplementedError or any
programmer error propagates. A Redis-down-at-enqueue is a platform-level
incident with its own monitoring; the model save is never 500'd for it.
(Note: Sidekiq 7 enqueues via redis-client, so the real connect error is
RedisClient::Error, notRedis::BaseError.)
Job-execution failures (Typesense down, a bad record): the job raises →
Sidekiq retries (retry: 3) → on each failure the global error handler fires
SlackSidekiqFailuresin production (config/initializers/sidekiq.rb) →
the job lands in the dead set (visible in the Sidekiq Web UI in every env).
Backstop: a nightly rake typesense:reindex (3:00am, config/schedule.rb)
rebuilds every collection from Postgres. It is off the critical path — it does
nothing in steady state — and exists to self-heal the drift the per-edit jobs
can't: jobs lost to a Redis flush, and records whose jobs exhausted retries.
There is no Sentry in this app (no gem; the initializer is commented out) —
ops alerting is Slack. Don't add Sentry.* calls expecting them to fire.
rake typesense:status — reindexing queue depth, retry/dead-set counts for
that queue, and each collection's live document count. Renders failures as
data (Redis or Typesense unreachable prints a status line, not a backtrace) so
it's usable during the incident you're diagnosing.
Sidekiq Web UI — queue depth, retries, and the dead set (basic-auth'd,
mounted in config/initializers/sidekiq.rb).
A growing dead set on :reindexing, or document_count far below the Postgres
source count, means sync is failing — check the dead set + run a reindex.
Cron is auto-managed.whenever is wired into Capistrano
(Capfile, whenever/capistrano), so cap deploy regenerates the crontab
from config/schedule.rb. The nightly typesense:reindex (3:00am) is applied
automatically — there is no manual crontab step, and no every-minute drain job
anymore (it was removed with the buffer table).
One-time cutover (only for the release that introduced direct-job sync).
Because the old drain cron was removed, any rows that were pending in the (now
dropped) typesense_sync_queue at cutover are not drained. Run a full reindex
once after that release deploys so the index reflects them:
Going forward this is not needed per-deploy — the per-edit jobs keep the index
fresh and the nightly reindex backstops drift.
The :reindexing queue must be worked. All sync jobs queue_as :reindexing; it is listed in config/sidekiq.yml and consumed by the
capistrano-sidekiq systemd unit. If you ever split Sidekiq processes by queue,
ensure a worker still covers :reindexing, or sync silently stops (the
failure class #1592 was originally about).
Health check after deploy / during an incident:rake typesense:status
(see §8) and the Sidekiq Web UI dead set.
Identity::SearchController#practitioners / #facilities query Typesense and
normalize each hit through normalize_practitioner / normalize_facility. Those
normalizers must handle two shapes, because the same method serves both the
Typesense path and the Postgres fallback:
Typesense returns location as a geopoint array[lat, lng].
The PG fallback (serialize_*_pg) returns location as a Hash.
Calling hit.dig('location', 'city') on the array raises
TypeError (no implicit conversion of String into Integer) — this 500'd doctor
search on staging the first time the index had real data (#1596). The fix is
location_hash(hit), which normalizes both shapes to a Hash before the
normalizer reads it. If you touch the search normalizers, keep both shapes
working (the regression spec domains/identity/spec/requests/search_typesense_geopoint_spec.rb
pins the array case).
Full reindex: bundle exec rake typesense:reindex (or
typesense:reindex_collection[practitioners|facilities]). Cross-tenant by
design — iterates every tenant and sets the GUC per tenant.
Add a new searchable field on role docs: add it to the collection's
serialize, and if a Facility/FacilityLocation column feeds it, add that
column to the relevant DENORMALIZED_TO_* list (§4) so changes propagate.
Add a new indexed model: include Shared::Concerns::TypesenseIndexable,
implement enqueue_typesense_sync(operation) to enqueue a model-specific job,
and have that job set the tenant context (§5) before re-fetching.
"Search results are stale": rake typesense:status (dead set? low doc
count?) → check the Sidekiq Web UI dead set for failed *Jobs → rake typesense:reindex to force-resync.
"Search returns nothing for a tenant after an edit": almost always a
missing GUC in a sync path — see row-level-security.md §6.1.
Sync jobs are unit-tested with the collection layer stubbed
(domains/identity/spec/jobs/typesense/*_spec.rb), pinning: the #1489 RLS
invariant (the job sets RlsContext.with(tenant_id) when there's no ambient
tenant), upsert-vs-delete dispatch, the delete-between-enqueue-and-run race,
and ReindexPractitionerJob's per-tenant isolation.
Model enqueue hooks are tested by invoking the private callback directly
(facility_typesense_sync_spec.rb, practitioner_typesense_sync_spec.rb,
professional_profile_spec.rb) — the suite uses DatabaseCleaner :transaction,
so after_commit doesn't fire on its own.
The transport-error guard contract is in
domains/shared/spec/services/typesense/enqueue_errors_spec.rb (swallows
RedisClient::Error; propagates programmer errors).
The read-side geopoint regression is in
domains/identity/spec/requests/search_typesense_geopoint_spec.rb.
The original sync used a Postgres buffer table (typesense_sync_queue) that
model after_commit hooks wrote to, drained by a scheduled Sidekiq job. That is
a database-as-queue anti-pattern when Sidekiq is already running, and its drain
was the single point that, when unscheduled, let the index silently drift (#1592
P0). It was replaced by direct Sidekiq jobs (#1660), the buffer table dropped
(#1661), and observability completed (#1662). The geopoint read crash was fixed
separately (#1596). Sidekiq's native retry + dead set + Web UI replace the
hand-rolled retry_count/dead_letter/queue_status the table used to carry.