Backend CI: local-first, GitHub only on the trunk

Backend CI runs the canonical gate — backend/bin/ci → backend/config/ci.rb (Rails 8.1 ActiveSupport::ContinuousIntegration). There is one definition of the checks; where it runs depends on the stage:

Everyday development does not spend GitHub-hosted minutes — the runner only fires on pull requests into master. The trade is intentional: run the suite on your own machine (it's the same bin/ci), and only pay for cloud CI at the trunk.

Why not a required status check / gh signoff? Both need GitHub branch protection, which this repo's plan doesn't offer (gh signoff install → HTTP 403). So the feature-PR gate is local discipline (bin/ci), and master is backstopped by the workflow.

Before opening a feature PRCopy Anchor Link

cd backend
bin/ci        # canonical gate — must be green before you open the PR

bin/ci runs, in order (every step runs even if one fails; non-zero exit if any failed):

1. bundle check || bundle install
2. bin/check-structure-sql-current — structure.sql drift guard
3. bin/check-no-domains-rspec — domains/ must stay Minitest-only
4. bin/rubocop-changed — RuboCop on files changed vs the base (blocking)
5. bin/rails db:test:prepare — load the test DB from structure.sql (RESETS it)
6. bin/ci-test-domains — domains/ Minitest suite, regression-gated on test/domains_failure_baseline.txt
7. pricing-catalog gating tests

Prereqs (same as any local dev): mise Ruby 3.4.9, PostgreSQL reachable per config/database.yml (with a test DB), Redis on localhost:6379.

Tests that use Redis must isolate their keys by process and example. Database isolation does not isolate Redis. The development notification outbox test uses Rails constant stubbing around each run to select a unique test key. It clears only that key and restores the constant, including after a failure. Do not clear the development outbox or flush a shared Redis database to prepare a test. This helper assumes the suite's process workers, not concurrent test threads.

Fast subset while iterating (skip the domains suite):

bin/check-structure-sql-current && bin/check-no-domains-rspec && bin/rubocop-changed

Postgres sizing on a many-core machineCopy Anchor Link

The domains suite forks one process per worker, each with its own kulcare_test-N database and connection pool. test/test_helper.rb caps the count at min(cores, 16) — not one per core — because two Postgres limits bite hard on a big box, and both fail in ways that look like a code catastrophe rather than a config ceiling:

The second one errors every single test (e.g. 8,217 errors out of 8,217 runs), so read the first error before concluding the branch is broken.

If you hit either, raise the limits (they are postmaster-level, so a restart is required; ALTER SYSTEM is revertible with ALTER SYSTEM RESET <name>):

ALTER SYSTEM SET max_locks_per_transaction = 2048;   -- default 64
ALTER SYSTEM SET max_connections           = 300;    -- default 100

sudo systemctl restart postgresql

For reference, on a 128-core box after tuning: the full domains suite runs in ~1m35s using 9 of 300 connections and peaking at 2 locks per backend. More than 16 workers bought nothing. PARALLEL_WORKERS=N still overrides the cap in either direction.

Faster local runs on macOS: the sharded domains suiteCopy Anchor Link

On macOS, forked parallel workers used to fail. The domains suite uses parallelize(with: :processes), and on this platform the forked workers could hang at 0% CPU or crash in libcurl (ethon), even with OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES. A serial run (PARALLEL_WORKERS=1) is reliable, but it takes about 25 minutes.

Why the forked workers failed (#2335)Copy Anchor Link

Both symptoms had one cause: the suite talked to the real Ably service.

1. A developer machine has config/master.key, so in test the credentials decrypt and carry ably_api_key. config/initializers/ably.rb then built a live ::AblyClient. CI has no master key, so its client was always nil.
2. ably-rest sends requests through Faraday's typhoeus adapter, which uses libcurl through ethon. Nothing calls curl_global_init before the fork, so the first test that published a realtime event called it inside a forked worker (ethon/curl.rb:61). On macOS, libcurl pulls in the Security and CoreFoundation frameworks, which are not safe to initialise in a forked child. The worker died: the Objective-C runtime aborted it, or, with OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES, it segfaulted.
3. A dead worker takes its test with it. Rails' parallel test server waits in at_exit for its queue to drain and for every worker to stop. It checks for dead workers only once, when that shutdown begins. So when workers die, the coordinator waits forever at 0% CPU. That is the exec_end_procs_chain hang.

The initializer now always leaves ::AblyClient as nil in the test environment, whatever the credentials say. Tests that need a client swap the constant with test/support/realtime_capture.rb. With that, nothing the domains suite runs reaches libcurl, and Ably behaves the same locally as on CI.

Keep libcurl out of forked workers. ably-rest loads typhoeus, so any HTTP client that picks its adapter automatically can end up on libcurl. Elasticsearch::Client is one: ELASTICSEARCH_CLIENT is still built in test on a machine with a master key, although no domains code uses it today. A test that needs HTTP should stub the client, as RealtimeCapture does for Ably.

This fix has not yet been confirmed on a Mac. If forked workers still fail there, use the sharded run below, and add the details to #2335. The earlier pg crash in forked workers is handled separately: test/test_helper.rb sets PGSSLMODE and PGGSSENCMODE to disable on macOS.

The sharded runCopy Anchor Link

To run the suite faster without forking, split it into shards:

cd backend
CI_DOMAINS_SHARDS=8 bin/ci          # the whole gate, with the domains step sharded
SHARDS=8 bin/ci-test-domains-sharded  # the domains suite only

How it works:

1. The script splits the domain test files round-robin across N separate rails test processes. It never forks them, and each one runs serially.
2. Each shard uses its own database, kulcare_test_s<i>, through DATABASE_URL. The databases load in parallel.
3. The RLS test-role grants run on one database at a time. The role is global to the cluster, and concurrent grants fail with "tuple concurrently updated".
4. The script prints one aggregate Minitest summary line, so the ratchet in bin/ci-test-domains works as before.
5. The script drops the shard databases when it finishes. Set KEEP_SHARD_DBS=1 to keep them.

Per-shard logs go to tmp/domains_shards/. The script fails when a shard database cannot be set up, or when a shard ends without a summary line. It never starts more shards than there are test files.

With 8 shards on a 16-core machine, the suite finishes in a few minutes and fails the same tests as a serial run. Use the same seed (SEED=51193) when you compare two runs by failing test name.

Isolated non-Billing checksCopy Anchor Link

Use a unique test database for each concurrent agent session. Do not run the standard shard script in two sessions at once: both runs use the same shard database names and log directory. Run each database's RLS grants in sequence. Apply those grants after the last schema load. A schema reload removes grants.

Test-mode reloading is disabled. Reloading while source files change replaces model classes and invalidates factory, association, and Mocha references.

See the non-Billing QA and storage repair guide for the scoped test command, deployment preflight, manual QA, and rollback. The storage migration refuses damaged UUID links. It does not recover them by guessing an owner or delete the underlying files.

The master gateCopy Anchor Link

.github/workflows/backend-ci.yml triggers on pull_request into master (paths backend/**) and runs the identical bin/ci on a GitHub-hosted Ubuntu runner with Postgres/Redis service containers. It's the last line of defense before the trunk — if your local bin/ci was green, this will be too.

Baseline ratchetCopy Anchor Link

test/domains_failure_baseline.txt is the domains non-passing ceiling. Keep it cloud-calibrated — read the domains non-passing: N line from a master runner job (the GitHub-hosted count runs a bit higher than local due to env differences) and ratchet down as red specs are fixed. The committed ceiling is zero. A baseline is a limit, not a current result. Use the latest complete CI run to determine whether the branch passes. Do not raise the ceiling to hide failures or substitute a passing subset for the complete gate.