ADR-0028 Background jobs & scheduling

Context

Some work must not run inside a request: sending email, calling a slow third party, building an export, anything that should be retried when it fails. Other work runs on a clock: cleanups, digests, syncs.

• The app is a Bun process in Docker (ADR-0003), replaced on every deploy and sometimes running as several replicas. A timer inside it fires once per replica and dies with the container
• Valkey already runs beside every app for rate limits and sessions, and ADR-0010 expected queues to land on it
• Postgres is the source of truth (ADR-0007), and ADR-0010 kept high-churn writes away from it on purpose

What is open: the queue library, where its state lives, and where jobs run.

Options considered

1. BullMQ on Valkey

Pros

• Queues, retries with backoff, delayed jobs, concurrency, and cron-style Job Schedulers in one MIT library
• Its docs list Valkey as a supported server
• Runs on ioredis, the client we already ship (a peer dependency since BullMQ 6.0)
• bullmq-otel links the enqueue and the processing of a job in one trace (ADR-0015)

Cons

• Needs a Valkey that persists and never evicts — ADR-0010 ran it without persistence
• Delivery is at-least-once: every handler must be safe to run twice
• An enqueue is not part of a Postgres transaction

2. A Postgres-backed queue (pg-boss, or BullMQ’s PostgreSQL backend)

Pros

• A job can be enqueued in the same transaction as the business write
• No change to Valkey; jobs are covered by the database backup

Cons

• Puts queue churn on Postgres — the load ADR-0010 kept out
• BullMQ’s Redis backend is its default; the PostgreSQL one is the newer path

3. Timers in the web process (setInterval, node-cron)

Pros: nothing to run. Cons: fires once per replica; lost on restart or deploy; no retries, no backoff, no record of what failed.

Decision

BullMQ on Valkey is the company standard for background jobs and scheduled work.

Concretely:

• Packages: bullmq 6.x (6.7.6 at time of writing) with ioredis 5.x installed explicitly — the same major ADR-0009 is held to
• Valkey: the instance from ADR-0010, now with AOF persistence (fsync every second) and maxmemory-policy noeviction. BullMQ’s production guide calls noeviction the only setting that guarantees correct queues
• Two connections, neither the limiter’s. Queues in the web process connect with enableOfflineQueue: false, so an enqueue fails fast when Valkey is down instead of holding the request. Workers connect with maxRetriesPerRequest: null, which BullMQ requires — on the limiter’s client that setting would make limit checks wait instead of falling back to the insuranceLimiter
• Jobs run in a worker container, never in the web process: a second entry point of apps/web in the same image, like the migration entry point of ADR-0016. The web process only enqueues. It becomes its own apps/worker/ (ADR-0026) only when it needs its own dependencies or image
• One module per queue, in server/jobs/: its name, a Zod schema for the payload (ADR-0005), and the handler. The payload is validated on enqueue and again in the worker — during a deploy an old version may have written it. Payloads carry IDs, not whole records
• Handlers are idempotent. A stalled or retried job runs again
• Every queue sets attempts with exponential backoff, and removeOnComplete / removeOnFail with a count or age — unbounded job history fills a store that cannot evict
• Scheduling is queue.upsertJobScheduler(id, { pattern }), called when the worker starts. It is an upsert, so replicas and redeploys do not duplicate a schedule. No node-cron and no setInterval for recurring work
• Enqueue after the Postgres commit, and design the job so that missing it or repeating it is safe
• Shutdown: on SIGTERM the worker calls worker.close() and exits within the grace period; a job that does not finish is marked stalled and picked up again
• Telemetry: bullmq-otel 2.x passed as telemetry to queues and workers, with enableMetrics on, so job traces and queue metrics leave through the app’s OTLP endpoint (ADR-0015)
• Tests: integration tests run real workers against the Valkey container of ADR-0014

The deciding factors: no new infrastructure, one library for queues and schedules, the Redis client we already use, and queue load kept off Postgres. We accept turning persistence on in Valkey and writing idempotent handlers.

Consequences

Easier:

• Slow and retryable work leaves the request path; a failed send is retried instead of lost
• Schedules live in code, deploy with the app, and run once however many replicas exist
• Pending verification links and OAuth state (ADR-0009) now survive a Valkey restart

Harder:

• Valkey needs a volume and a memory limit with headroom. Queues, sessions, and rate limits share it: a queue that fills memory makes session and limiter writes fail
• Nodus must run the worker entry point as a second container — not yet part of the contract in ADR-0016
• AOF protects against a restart, not the loss of the host; whether queue contents need a backup is decided per deployment (ADR-0032)
• Idempotency is a habit, enforced in review

Given up: enqueueing a job in the same transaction as a database write. Accepted — one store for queues, counters, and sessions is worth designing jobs that are safe to miss or repeat.