What this is
Swappr has a dedicated worker process (@swappr/worker) that runs separately from the API. It consumes four BullMQ queues for the heavy / scheduled / long-running work that should not block API request handlers. This page covers:
- The four queues and what they do.
- How both the producer (api) and consumer (worker) sides flip between in-memory stubs and BullMQ on the same
REDIS_URL signal.
- Scheduling — what runs on cron, what reacts to enqueue.
- The graceful shutdown order.
- The stub-mode behaviour for
pnpm dev and unit tests.
- The deferred-at-credentials matrix.
Source of truth: apps/worker/src/index.ts (bootstrap), apps/api/src/index.ts (producer selection), SWAPPR_BACKEND_PLAN §13 + §14.
The four queues
| Queue | Producer | Consumer | Scheduling | Side effect |
|---|
push-delivery | api (send-message, match handshake, warn) | worker — push-delivery.ts | Reacts to enqueue | FCM HTTP v1 send → user devices. Throttle store per-recipient. |
retention-cleanup | Scheduler (BullMQ repeat) + manual trigger | worker — retention-cleanup.ts | 03:00 UTC daily + manual | S3 deleteObject(fileKey) for tenancy documents 30 days post-decision; soft-delete uploads row. |
gdpr-export | api (GET /me/export) | worker — gdpr-export.ts | Reacts to enqueue | Collect user data → ZIP → S3 PUT private bucket → mint 24h URL → email user. |
gdpr-erasure | Scheduler (BullMQ repeat) + manual trigger | worker — gdpr-erasure.ts | 04:00 UTC daily + manual | Hard-delete child rows for soft-deleted users >30d; tombstone the user row (PII nulled, userErasedAt set); anonymize audit_logs. |
Bootstrap diagram
REDIS_URL — one signal, two sides
The producer and consumer both check the same env var:
- API process (
apps/api/src/index.ts) — resolvePushQueue, resolveRetentionEnqueuer, resolveGdprExportQueue, resolveErasureEnqueuer all key off process.env.REDIS_URL. With it: BullMQ-backed producer. Without it: a no-op enqueuer (returns Promise.resolve() — the API still returns 202 Accepted so the surface looks identical to the caller, but no work happens).
- Worker process (
apps/worker/src/index.ts) — bootWorker({ redisUrl }). With it: constructs all four Worker instances + a shared ioredis connection + schedules the two repeatable jobs. Without it: returns a WorkerHandle in 'no-worker' mode and exits the bootstrap path with a warn log.
This single signal means dev / unit tests just work with no Redis — the entire surface is exercisable, only the actual side-effects (FCM send, S3 delete, ZIP upload, hard delete) are skipped.
The api-side selector is required to inject a queueFactory when REDIS_URL is present (production caller must construct the BullMQ Queue). The selector deliberately throws if REDIS_URL is set but no factory is supplied — fail loud, not silently fall back to no-op when ops thinks BullMQ is wired.
Scheduling — cron vs reactive
| Queue | Trigger |
|---|
push-delivery | Reactive only. Enqueued on every message send, match handshake, and moderator warn. No cron. |
retention-cleanup | Cron + manual. scheduleRetentionCleanup registers a BullMQ repeatable job at 03:00 UTC daily (idempotent — BullMQ dedups on jobId). Manual: POST /admin/retention/run. |
gdpr-export | Reactive only. Enqueued per-user when they hit GET /me/export. |
gdpr-erasure | Cron + manual. scheduleGdprErasure registers a BullMQ repeatable job at 04:00 UTC daily. Manual: POST /admin/erasure/run. |
Graceful shutdown
On SIGTERM or SIGINT the worker process closes resources in order:
- Workers (stop accepting new jobs; let in-flight jobs finish to BullMQ’s
lockDuration ceiling).
- Queues (drain producer-side state).
- Shared ioredis connection (
.quit() — graceful FIN, not .disconnect()).
shutdown = async () => {
await Promise.all([pushWorker.close(), retentionWorker.close(),
gdprExportWorker.close(), gdprErasureWorker.close()]);
await Promise.all([retentionQueue.close(), gdprErasureQueue.close()]);
await redis.quit();
};
Stub-mode behaviour
When REDIS_URL is absent:
| Surface | Behaviour |
|---|
| API endpoints | All return the same status code as production (e.g. /me/export → 202). The DB row IS still inserted. Only the BullMQ enqueue is replaced with a Promise.resolve(). |
| Worker process | If you boot it without REDIS_URL, it logs "REDIS_URL not set — running in no-worker mode" and returns a WorkerHandle with mode: 'no-worker'. No Worker instances are constructed. |
| Tests | Unit tests inject in-memory fakes via the factory overrides. They never touch Redis or BullMQ. |
pnpm dev with no Redis installed and the entire HTTP surface works — the only thing missing is the asynchronous side-effects (push notifications, retention cleanup, export ZIP generation, erasure tombstoning).
Deferred at credentials
These are wired up but gated on a production credential that isn’t in Phase 7’s scope:
| Queue | Deferred dep | What’s missing | Effect |
|---|
push-delivery | Real FCM service account JSON | Currently the push sender is stubbed when FCM_* env is absent; the queue still runs but the PushSender.send() returns PUSH_NOT_CONFIGURED. | Push attempts are logged + dropped. The DB still records who would have received what. |
retention-cleanup | S3 DELETE permission on the tenancy bucket | If the bucket IAM policy denies DELETE, deleteObject throws. | Per-row try/catch logs + continues. The row stays deletedAt === null → retried next day. |
gdpr-export | S3 PUT permission on the private bucket + Resend creds | Without PUT, the worker fails the job (FAILED). Without Resend, the job stays READY but the user gets no email — they recover via GET /me/export/:jobId. | See per-queue behaviour. |
gdpr-erasure | S3 DELETE permission on the private + uploads buckets | Same as retention — per-user try/catch swallows. | User stays soft-deleted → retried next day. |
BILLING_NOT_CONFIGURED) until the real RevenueCat secret is wired.
Why a separate process?
Three reasons (BACKEND_PLAN §13):
- Blast radius. A worker crash from a malformed FCM token, a runaway ZIP, or an S3 hang must not take the API down.
- Scaling profile. The API scales on request volume (spiky). The worker scales on queue depth (smoother). They need different deployment knobs.
- Deploy cadence. Worker bug fixes (e.g. a new push payload field) can ship without an API redeploy.
See also
