ADR-0016 CI/CD

Context

Our repositories live on GitHub or on our own Forgejo, and an app may move between them. Both forges run Actions-style workflows, but they are not identical:

• Forgejo Actions aims for familiarity, not compatibility. It reads .forgejo/workflows/, falling back to .github/workflows/ only when .forgejo/workflows/ doesn’t exist. Short uses: references resolve against https://data.forgejo.org (a mirror of common actions), not GitHub. Job-level permissions and continue-on-error are ignored, and OIDC uses enable-openid-connect instead of permissions: id-token: write
• GitHub only accepts owner/repo@ref in uses: — no full URLs — so a third-party action that isn’t mirrored on data.forgejo.org can’t be referenced the same way on both
• Runners differ: GitHub’s hosted ubuntu-latest is a fresh VM per job with Docker; Forgejo Runner runs jobs in a container (Debian with Node by default) on a long-lived machine, and Docker inside jobs needs runner configuration

Current versions: Forgejo 17.0 (stable), 15.0 LTS (supported until 2027-07-15); Forgejo Runner 13.2.

Deployment is done by Nodus, our proprietary platform. This ADR covers everything up to a published image, plus the contract that image must meet for Nodus. How Nodus is triggered, and how it promotes and rolls back, is defined by Nodus.

What CI must run is already decided: bun run check, bun test for unit and integration tests, and Playwright E2E (ADR-0035, ADR-0014). (Amended 2026-11-13: this ADR first installed Vite+ and Node and ran on Debian slim; ADR-0035 moved the toolchain to Bun, and the runtime image is now Bun’s distroless image, run as a non-root user.)

Options considered

Workflows across two forges

1. One portable workflow set in .github/workflows/

Pros: one file to maintain; Forgejo reads it through its fallback. Logic lives in bun run scripts, so the YAML stays thin. Cons: limited to the common subset — no third-party actions beyond those mirrored on data.forgejo.org, no forge-specific features.

2. Two workflow sets (.github/workflows/ and .forgejo/workflows/)

Pros: each forge uses its best features (e.g. oven-sh/setup-bun on GitHub). Cons: two pipelines that drift; every change is made twice.

3. One forge only

Pros: no compatibility work. Cons: doesn’t match where our repositories actually live.

Installing the toolchain in CI

• oven-sh/setup-bun — the official action, with caching, and it reads packageManager; but it is a third-party action, so on Forgejo it resolves only if data.forgejo.org mirrors it
• oven/bun as the job container — the official image; but jobs that need Docker (Testcontainers, E2E, image builds) would need the host’s Docker socket mounted into it
• The official installer in a run: step (curl -fsSL https://bun.sh/install | bash -s bun-v<version>) — plain shell, identical on both forges

Isolating Forgejo jobs

Forgejo Runner executes whatever a workflow says — remote code execution by design. GitHub-hosted runners already give every job a fresh VM; on Forgejo, that isolation is ours to build.

• Persistent runner in containers, Docker-in-Docker — the default setup; but containers share the host kernel, and state (caches, images, leftover files, credentials) survives from one job to the next
• LXC runner labels — closer to a VM, still a shared kernel
• One ephemeral Firecracker microVM per job — hardware-virtualized isolation with its own kernel and its own Docker engine, booted in seconds from a clean image and destroyed after the job. Forgejo supports this model natively: an ephemeral runner runs at most one job before Forgejo deletes its registration, enforced server-side. Needs KVM hosts and an orchestrator we run

Decision

One portable workflow set in .github/workflows/, running on both GitHub Actions and Forgejo Actions, builds one immutable image per commit to the forge’s own registry; Nodus deploys it. This is the company standard for all new apps.

Workflows

• Only .github/workflows/. Never add .forgejo/workflows/ — its presence makes Forgejo ignore .github/workflows/

• The common subset only: runs-on: ubuntu-latest (our Forgejo runners register that label), github.* contexts (Forgejo provides them), and actions/checkout as the only action. Anything else is a run: step calling bun

• Actions pinned by full commit SHA with the tag in a comment. data.forgejo.org mirrors upstream commits, so one SHA resolves on both (verified for actions/checkout v7.0.1)

• Bun installed by its official installer at the exact version in the root package.json’s packageManager field (ADR-0035), then bun ci, which fails if bun.lock is out of date

• Triggers: pull_request and push to main. concurrency per ref, cancelling superseded runs on pull requests only

• Jobs:

  Job	Runs (after bun ci)	Needs Docker
  check	bun run check, bun run test	No
  integration	bun run test:int	Yes
  e2e	bunx playwright install --with-deps chromium, bun run build, bun run test:e2e	Yes
  image	on main only, after all three pass: build, push, upload source maps, hand off to Nodus	Yes

• main is protected on both forges: changes land through pull requests, with check, integration, and e2e as required checks

• Caching: Bun’s install cache only, keyed on bun.lock, saved from main runs only — a pull request can’t poison a cache that main later trusts

Security

• No pull_request_target. Pull-request runs from forks get no secrets on either forge; keep it that way
• actions/checkout with persist-credentials: false, so later steps can’t read the token from disk
• Least privilege: permissions: contents: read at the top of every workflow; only the image job uses the registry secret. Forgejo ignores job-level permissions, so on Forgejo, isolation comes from secrets being available only on main
• Registry credentials come from secrets.REGISTRY_TOKEN, scoped to package write; the registry and image name come from vars.REGISTRY and vars.IMAGE, so the same YAML pushes to GHCR or to Forgejo’s container registry

Forgejo runners: one ephemeral Firecracker microVM per job

• Every Forgejo job runs in its own Firecracker microVM (Firecracker 1.x, on KVM hosts), which is destroyed when the job ends. Nothing survives between jobs: no files, images, caches, or credentials. This matches GitHub-hosted runners, where each job also gets a fresh VM
• Ephemeral registration: the pool manager registers each VM’s runner as ephemeral through Forgejo’s API, so Forgejo hands it at most one job and then deletes the registration. The runner inside the VM exits after that job; the pool manager kills the VM, deletes its writable disk, and boots a replacement
• Warm pool: the pool manager keeps a small number of booted, registered VMs waiting, so jobs start without waiting for a boot. Pool size per host is set by CPU and memory, not by demand peaks — jobs queue when the pool is busy
• Clean image: each VM boots from a read-only golden root filesystem (Ubuntu with Docker Engine, git, curl, unzip — which Bun’s installer needs — and bash) plus a fresh, empty writable layer. The golden image and guest kernel are built by CI, versioned, and rebuilt weekly and on security advisories. The guest kernel includes what Docker needs (overlayfs, cgroups v2, netfilter)
• Inside the VM, steps run directly on the VM (ubuntu-latest:host label) with the VM’s own Docker Engine — so Testcontainers, E2E, and image builds work exactly as on GitHub. The VM is the isolation boundary; the host’s Docker, if any, is never reachable
• Network: each VM gets its own tap device behind NAT. Egress to the internet is allowed (npm, GHCR, data.forgejo.org, bun.sh and the GitHub releases it downloads from); private address ranges are blocked except the Forgejo instance, the cache server, and the registry mirror. VMs can’t reach each other, the host, Nodus, or production
• Shared services live outside the VMs: the Actions cache server and a pull-through registry mirror (Docker Hub, GHCR) run on the host network, so ephemeral VMs still get warm Bun caches and fast image pulls. The cache server keeps Forgejo Runner’s rule that pull_request runs can’t write caches other events read
• Secrets exist only inside a job’s VM and vanish with its disk. The registration token is single-use

The image (contract with Nodus)

• One image per commit, tagged with the full git SHA, never latest. OCI labels record the source repository and revision. The same image is promoted through every environment — never rebuilt per environment
• Two stages, both Bun’s official images at the version in packageManager, pinned by digest:
  • Build on oven/bun:<version>: bun ci, bun run build (Nitro’s bun preset writes the server to .output/), and bun build --target bun for the other entry points — instrumentation.ts, the migration runner, and the worker (ADR-0028)
  • Runtime on oven/bun:<version>-distroless: no shell, no package manager, no curl. It gets .output/, the bundled entry points, and the drizzle/ migrations folder — nothing else from the build stage. Bun.SQL is built in (ADR-0008), so no stage needs a compiler or libpq
• Runtime stage settings:
  • USER 65532:65532 — the distroless Bun image runs as root unless told otherwise. The app writes nowhere but /tmp, so the container can run with a read-only root filesystem
  • NODE_ENV=production
  • ENTRYPOINT ["/usr/local/bin/bun"] and CMD ["--preload", "./instrumentation.js", ".output/server/index.mjs"] (ADR-0015); the migration runner and the worker are the same image with a different CMD, which keeps the --preload
  • HEALTHCHECK in exec form, since there is no shell: bun -e with a fetch to /healthz that exits non-zero on failure
• Source maps are uploaded to GlitchTip by the image job and never copied into the runtime stage (ADR-0015)
• Configuration is environment variables only, injected by Nodus at runtime and validated with Zod at boot (ADR-0005). No secrets in the image, its build arguments, or its layers
• Health: GET /healthz and GET /readyz as defined in ADR-0015; the contract expects Nodus to send traffic only to instances whose /readyz passes
• Graceful shutdown on SIGTERM: stop accepting connections, finish in-flight requests, close the Postgres pool and Valkey client, flush the OTel SDK, then exit within Nodus’s grace period
• Migrations ship in the image as a separate entry point using Drizzle ORM’s runtime migrator (Drizzle Kit is a dev dependency and isn’t in the image). Nodus runs it once per deploy, before the new version takes traffic
• Migrations are backward-compatible: during a deploy the old and new versions run side by side, so every migration must work with the previous release — expand first (add nullable columns, new tables), contract in a later release (drop what the old version used)

Dependency updates

• Renovate, self-hosted, on both forges (it supports GitHub and Forgejo; Dependabot is GitHub-only). It updates npm packages, action SHAs, and base-image digests. Packages that must move together are grouped: packageManager with the oven/bun image digests (ADR-0035), TanStack Query/Router/Start (ADR-0012), the four Better Auth packages (ADR-0009), and @opentelemetry/* (ADR-0015)

The deciding factors: one pipeline regardless of which forge hosts the repository, every security invariant from earlier ADRs gated before merge, and an image contract that lets Nodus deploy any app the same way. We accept the common-subset limit on workflows and building and running the microVM runner pool.

Consequences

Easier:

• Moving a repository between GitHub and Forgejo needs no pipeline changes — only variables and secrets
• A failing guard test (ADR-0014) blocks the merge, not production
• Every deployed artifact is traceable to one commit and identical in every environment; rollback is redeploying an earlier SHA
• Nodus deploys every app the same way: same health endpoints, same shutdown behavior, same migration entry point
• A malicious or broken job on Forgejo can’t affect the next one: its VM, disk, Docker state, and secrets are gone when it ends
• Jobs on both forges see the same environment — a fresh VM with Docker — so a workflow that passes on one passes on the other

Harder:

• The common subset rules out most marketplace actions; new steps are shell around bun, Docker, or a CLI
• The installer step trusts bun.sh at run time; the version is pinned but the script isn’t. Review if oven-sh/setup-bun becomes available on data.forgejo.org
• A distroless image has no shell to debug in: inspect a running container with docker debug or an ephemeral debug container, never by switching the base image
• The microVM pool is ours to build and operate: KVM hosts (bare metal, or nested virtualization), the pool manager, the golden image and guest kernel, the cache server, the registry mirror, and capacity
• Every Forgejo job starts cold: Bun, dependencies, and Playwright browsers install each time, softened only by the external cache and registry mirror. Measure job times; bake slow, version-stable pieces into the golden image only when they don’t vary per project
• Expand/contract migrations take two releases for any destructive schema change
• The Nodus side of the contract (trigger, environments, promotion, rollback, secrets injection) isn’t recorded here; it must match these assumptions

Given up: setup-bun and the GitHub marketplace on GitHub-hosted repositories, forge-specific pipeline features, and building per environment. Accepted — one pipeline on both forges and one artifact per commit matter more than convenience on either forge.