ADR-0030 File & object storage

Context

Apps store files: uploads, generated exports, attachments. They need a home that is neither a Postgres row nor the container’s disk.

• Containers are replaced on every deploy and replicas do not share a filesystem (ADR-0003)
• The edge caps request bodies at 10 MB (ADR-0017), so large uploads cannot pass through the app
• Deployments differ: Ceph on our own infrastructure, or an S3-compatible store a client already runs
• Files carry access rules: who may read a file is decided by the app, per request (ADR-0009)

What is open: the storage API, the client, and how browsers send and fetch files.

Options considered

1. Local disk or a Docker volume

Pros: nothing extra to run. Cons: tied to one host, not shared between replicas, and backup is ours to invent.

2. Postgres (bytea or large objects)

Pros: transactional with the rest of the data; one backup. Cons: every byte streams through the app and the connection pool; the database and its backups grow with file volume.

3. The S3 API against an S3-compatible store

Pros

• One API whichever store the deployment has
• Presigned URLs let the browser upload to and download from the store directly, past the app and its body limit
• Ceph’s RADOS Gateway implements what we need: presigned URLs, bucket CORS, multipart uploads, lifecycle rules

Cons

• A store to run — and Ceph is not small. That belongs to the deployment
• “S3-compatible” is not identical: checksum defaults and bucket addressing differ between stores

4. A provider’s own SDK (Azure Blob, Google Cloud Storage)

Pros: first-class features of that provider. Cons: a storage module per provider; moving a product means rewriting it.

Decision

The S3 API is the company standard for files. Ceph RADOS Gateway is the default store on our infrastructure; a deployment may use any S3-compatible store.

Concretely:

• Client: Bun’s built-in S3Client (import { S3Client } from "bun", ADR-0035), constructed from endpoint, region, credentials, and bucket declared in server/env.ts (ADR-0026). Never the implicit Bun.s3 client, which reads S3_* and AWS_* variables on its own. (Amended 2026-11-13: this ADR first chose @aws-sdk/client-s3 with its presigner; Bun’s client replaces both.)
• Path-style addressing, the client’s default (virtualHostedStyle stays false), so any S3-compatible endpoint works without wildcard DNS
• One storage module, src/server/storage/, is the only code that imports S3Client. Feature code receives it as a parameter (ADR-0027) and uses nothing a plain S3 API does not have
• Buckets are private. No public-read ACLs or policies
• A files table in Postgres is the source of truth (ADR-0008): key, size, content type, owner, status. The store holds bytes; every access decision reads the row
• Uploads go from the browser to the store. A POST server function checks the session, validates the declared type and size with Zod, picks the key, and returns a short-lived presigned PUT URL (presign(key, { method: "PUT", expiresIn, type })). The server chooses keys ({owner}/{uuid}) — never the client’s file name
• Confirm before trusting. After the upload the server reads the object’s metadata with file(key).stat(), compares size and type with what was declared, and deletes it on a mismatch. Only then is the row marked ready
• Downloads are short-lived presigned GET URLs, issued after the same authorization check. Untrusted types are served with Content-Disposition: attachment (the contentDisposition presign option)
• Deployment setup: the store’s endpoint is reachable by browsers over TLS, with CORS allowing the app’s origin, and lifecycle rules that abort incomplete multipart uploads and expire unconfirmed objects
• Durability and backup of the store are decided per deployment, like the database (ADR-0032)
• Tests: unit tests pass a fake storage module; integration tests run against an S3-compatible container in Testcontainers (ADR-0014)

The deciding factors: one API across every deployment, uploads that never pass through the app or its body limit, and files that stay behind the app’s own authorization. We accept running a store and checking each store’s S3 dialect against the calls we use.

Consequences

Easier:

• Replicas and deploys never lose a file; the app stays stateless
• Large uploads work without raising the edge’s body limit
• Moving between Ceph and another S3-compatible store is a configuration change

Harder:

• An upload is two steps and can stop halfway: unconfirmed rows and objects need cleanup
• The store’s endpoint, CORS, and lifecycle rules are deployment work that must exist before the first upload. Bun’s client covers object calls — presign, stat, read, write, delete, list — not bucket administration, so the deployment sets those with its own tools
• A file and its row can disagree; the row wins, and orphans are swept

Given up: store-specific features (event notifications, object lock, vendor CDNs) as a standard. Accepted — a product that needs one writes its own ADR.