ADR-0015 Observability

Context

We self-host on Docker (ADR-0003), so nobody else sees our requests, errors, or slow queries. Every request crosses several hops — Start middleware (rate limiter, CSRF), server functions or API routes, Better Auth, Valkey, Postgres — and a problem in any of them looks the same to a user: slow or broken.

We need:

• Traces across those hops, so a slow request points at its slow query or Valkey call
• Metrics for dashboards and alerts: request rate, errors, latency, runtime health, 429s
• Logs that link to the trace they came from
• Browser errors with readable stack traces — the server never sees a crashed component
• Health endpoints for Docker and the reverse proxy, which ADR-0003 left to us

Where telemetry is stored, viewed and alerted on is outside the app. The platform gives each app an OTLP endpoint, and whatever runs behind it can change without the app knowing. This ADR covers what the app emits, what it must never emit, and where browser errors go.

Constraints:

• Vendor-neutral instrumentation: apps should not change when the backend does
• Personal data: telemetry carries user IDs and URLs; magic-link and verification tokens travel in query strings (ADR-0009). What we record is a privacy decision, and the app can’t count on anything downstream to filter it
• Every hop must be traced: OpenTelemetry has maintained instrumentations for ioredis, pino, and node:http, and BullMQ and TanStack AI emit spans themselves. Bun’s HTTP server, Bun.SQL, and Bun’s fetch have no instrumentation yet (ADR-0035), so those spans are ours to make

Options considered

Instrumentation

• OpenTelemetry — the vendor-neutral standard; one SDK for traces, metrics, and logs; every mainstream backend accepts OTLP. Apache-2.0
• A vendor SDK (Sentry, Datadog) — less setup, but it ties every app to that vendor; Sentry’s server SDK runs its own OTel setup

Where sensitive data is filtered

1. In a Collector beside the app

Pros: one place for every service on a host, the reverse proxy included; the app stays plain. Cons: the app’s privacy depends on infrastructure outside it; an app exporting anywhere else — a laptop, a new platform — sends tokens as they are.

2. In the app, before export

Pros: nothing sensitive leaves the process, whatever receives it; a test in the app’s own suite proves it. Cons: every app carries the filter in its instrumentation.ts; the SDK hook it uses is still experimental.

Browser errors

• GlitchTip + Sentry browser SDK — self-hosted, MIT; accepts the Sentry SDKs’ protocol; source maps; runs on Postgres with optional Valkey. No session replay
• Sentry SaaS — the best grouping and replay, but per-event pricing and data off our infrastructure
• OpenTelemetry only — browser errors would go untracked

Decision

OpenTelemetry in every app, exported over OTLP to the endpoint the platform provides, with sensitive data filtered in the app before export; GlitchTip for browser errors. Storage, dashboards and alerts are outside the app and not decided here. This is the company standard for all new apps.

In every app

• OTel Node SDK (@opentelemetry/sdk-node) starts before the server: the Docker image starts the app with bun --preload ./instrumentation.js (ADR-0016), and that file starts the SDK. Nothing app-specific runs before it. Bun has no ESM loader hooks (module.register is a no-op), so instrumentations patch through require-in-the-middle only. (Amended 2026-11-13: this ADR first used NODE_OPTIONS="--import …" on Node; ADR-0035 moved the runtime to Bun.)

• Every entry point preloads it: the web server, the worker (ADR-0028), and the migration runner, so jobs and migrations are traced like requests

• Instrument every hop a request or job takes, with a maintained instrumentation where one works on Bun, and our own code where none exists:

  Hop	Spans and metrics from
  Incoming requests	Ours: trace, the first middleware in src/start.ts — SERVER span {method} {http.route} for pages, server functions, and API routes alike, incoming traceparent, http.server.request.duration
  Server functions	Ours: traceServerFn, the global function middleware (functionMiddleware in src/start.ts) — an INTERNAL span per call named after the function (serverFnMeta.name), with code.function.name and code.file.path. Route loaders call server functions, so their server work lands here
  Postgres (Bun.SQL)	Ours: traceSql wraps the client Drizzle gets — a CLIENT span per query, also inside transactions, and db.client.operation.duration
  Valkey (ioredis)	@opentelemetry/instrumentation-ioredis, with dbStatementSerializer set to the command name only: keys hold session tokens, IPs, and emails
  Outgoing fetch	Ours: instrumentation.ts wraps globalThis.fetch — a CLIENT span per call, traceparent injected, http.client.request.duration; Bun-only init options such as proxy and tls pass through untouched
  Outgoing node:http and node:https	@opentelemetry/instrumentation-http, for any dependency that skips fetch; the instrumentation works on Bun since 1.4
  Jobs	bullmq-otel: enqueue and processing in one trace, plus queue metrics (ADR-0028)
  AI calls	TanStack AI’s OTel middleware: run, model turn, and tool spans, token metrics, content capture off (ADR-0025)
  Logs	@opentelemetry/instrumentation-pino: records go to the OTel logs pipeline with trace_id and span_id
  Object storage, SMTP	Ours: withSpan(name, fn) from src/server/telemetry.ts, around each call in server/storage/ and server/mail/ — Bun’s S3Client and Nodemailer have no instrumentation
  Runtime	@opentelemetry/instrumentation-runtime-node, kept only if the first app shows real event-loop and heap values on Bun; process CPU and memory come from container metrics either way

• Not instrumented, on purpose: fs, dns, and net (a span per file read or socket is noise in a web app), and the browser (exporting there needs a public OTLP endpoint; GlitchTip already covers browser errors). @8monkey/opentelemetry-instrumentation-bun-sql exists, but 0.2.0 pins @opentelemetry/core 2.x while our SDK is on 3.x

• Our four pieces — trace, traceServerFn, traceSql, and the fetch wrapper — follow the semantic conventions, record db.query.text exactly as Drizzle sends it (placeholders, never parameter values), and are covered by an integration test: one span per server function call, one per query, also inside a transaction, one per fetch, and no parameter value or query string on any of them. Each carries a shortcut: comment: delete it once Bun.otel (oven-sh/bun#39965) or Start’s built-in tracing covers the same hop

• Start’s own guide: trace and traceServerFn follow the request- and function-middleware patterns in TanStack Start’s Observability guide, with two changes: spans are named by route pattern and function name, never by raw path, and no span records a full URL. The guide calls its OpenTelemetry setup experimental and announces built-in instrumentation for server functions, middleware, and route loaders; when that ships, it replaces trace and traceServerFn. Its in-memory metrics endpoint and verbose health check are not used — metrics go through OTel, and /healthz and /readyz expose nothing

• @opentelemetry/* packages move together — the SDK packages release in lockstep

• Configuration through standard OTEL_* variables: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT (set by the platform; the app’s contract ends there), OTEL_RESOURCE_ATTRIBUTES with deployment.environment.name and service.version set to the git SHA. Sampling: parentbased_traceidratio, starting at 1.0 and lowered per product when volume demands it

• No query string leaves the process. The trace middleware records url.path, never url.full or url.query. Behind it, a span processor cuts the query from every URL attribute in its onEnding hook, which runs before any exporter sees the span, so a URL a library or a later change records is cut too:

  // apps/web/instrumentation.ts (excerpt)
  import { NodeSDK } from "@opentelemetry/sdk-node";
  import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
  import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
  import { HttpInstrumentation } from "@opentelemetry/instrumentation-http";
  import { IORedisInstrumentation } from "@opentelemetry/instrumentation-ioredis";
  import { PinoInstrumentation } from "@opentelemetry/instrumentation-pino";
  import { RuntimeNodeInstrumentation } from "@opentelemetry/instrumentation-runtime-node";
  
  // Tokens travel in query strings (ADR-0009): cut them before any exporter sees a span.
  const URL_KEYS = ["url.full", "http.url", "http.target"];
  const dropQueries = {
    onStart() {},
    onEnding(span) {
      for (const key of URL_KEYS) {
        const value = span.attributes[key];
        if (typeof value === "string") span.setAttribute(key, value.split("?")[0]);
      }
      if ("url.query" in span.attributes) span.setAttribute("url.query", "");
    },
    onEnd() {},
    forceFlush: async () => {},
    shutdown: async () => {},
  };
  
  new NodeSDK({
    spanProcessors: [dropQueries, new BatchSpanProcessor(new OTLPTraceExporter())],
    instrumentations: [
      new HttpInstrumentation(),
      // Keys hold session tokens, IPs and emails (ADR-0009, ADR-0010): record the command only.
      new IORedisInstrumentation({ dbStatementSerializer: (command) => command }),
      new PinoInstrumentation(),
      new RuntimeNodeInstrumentation(),
    ],
  }).start();
  
  // ...then the globalThis.fetch wrapper, so every library that loads later gets it.

  onEnding is experimental in the SDK, so a guard test (ADR-0014) sends a request carrying ?token=canary through the instrumented server, with an in-memory exporter, and fails if any span holds canary

• Route names, never raw URLs: the trace middleware sets http.route to the matched route pattern, API routes included (ADR-0036). Raw paths with IDs would explode metric cardinality

• Logging with pino 10: one logger module per app, JSON to stdout (so docker logs still works) and to the OTel logs pipeline through @opentelemetry/instrumentation-pino, which also stamps trace_id and span_id on every record. pino-pretty in development only

• Every request has a request ID, set by the middleware right after trace in src/start.ts — before the rate limiter and CSRF check, so rejected requests carry one too (requestMiddleware: [trace, requestId, rateLimit, csrf, apiCaller], functionMiddleware: [traceServerFn]):

  • Reuse the incoming X-Request-Id header when present and well-formed: 1–128 characters of A–Z a–z 0–9 . _ : -. Otherwise generate one with crypto.randomUUID(). A malformed value is replaced, never logged as-is — the header is client-controlled, so validating it prevents log injection and oversized fields
  • Held in an AsyncLocalStorage for the request. The logger’s pino mixin reads it, so every log line during the request carries request_id without passing it around. It is also set as the request_id attribute on the server span, so a request ID found in a log or support ticket leads to the trace
  • Returned in the X-Request-Id response header (set on result.response after next()), and shown with the generic error message, so users and support can quote it
  • One ID per request: API routes read the same store. Calls to our own services forward X-Request-Id; calls to third parties don’t. traceparent goes on every outgoing call — it holds only random IDs

• Never record secrets: pino redact covers password, token, secret, authorization, cookie, and set-cookie; never log request bodies on auth endpoints; never enable HTTP header capture for cookie, authorization, or set-cookie. Record user IDs, never emails or names

• Errors on the server are recorded on the active span (recordException, status ERROR) and logged at error with the trace ID; the user sees a generic message (ADR-0013). Server errors go out over OTel with the rest of the server’s telemetry, not to GlitchTip — one server-side instrumentation

• Business metrics through the OTel metrics API (@opentelemetry/api meter), including a counter of rate-limited requests by scope (ADR-0010)

• Health endpoints as Start server routes outside /api/*, so the limiter skips them (ADR-0010) and they are never traced:

  • GET /healthz — liveness: the process answers; checks nothing else. Docker’s HEALTHCHECK uses it
  • GET /readyz — readiness: Postgres SELECT 1 and Valkey PING. The reverse proxy routes traffic only to ready instances (ADR-0017)

• Browser errors: @sentry/react 11.x, initialised on the client only, DSN pointing at GlitchTip. release is the git SHA; sendDefaultPii: false; tracesSampleRate: 0 (traces belong to OTel); query strings scrubbed in beforeSend and beforeBreadcrumb. Source maps are uploaded to GlitchTip in CI and removed from the image, so they are never served publicly

Beyond the app

• GlitchTip self-hosted with its own Postgres database — never an app’s database. Events are kept 90 days
• Where OTLP goes, how long telemetry is kept, dashboards and alerts belong to whoever runs the backend, not to the app. The app’s part is the signals: route-named HTTP metrics, query and outgoing-call durations, queue metrics, /readyz, and the rate-limit counter — enough to alert on 5xx rate, p95 latency, slow queries, a growing queue, a failing /readyz and a jump in 429s. Postgres and Valkey metrics are collected outside the app too

The deciding factors: vendor-neutral instrumentation that survives a backend change, every hop of a request or job in one trace — by maintained instrumentation where it works on Bun, by four small pieces of ours where it doesn’t — privacy enforced inside the app rather than trusted to whatever receives its telemetry, and browser errors kept on our infrastructure. We accept carrying the filter and the wrappers in every app, and running GlitchTip.

Consequences

Easier:

• A slow request or job opens as one trace: middleware, server function or API route, every Postgres query and Valkey command, outgoing calls, model turns, and the jobs it enqueued
• Logs, traces, and metrics link to each other through trace_id, in whatever backend receives them
• A user’s or support ticket’s request ID finds every log line of that request
• Browser crashes arrive with readable stack traces, grouped by release
• A token in a URL never leaves the process, wherever the app exports to — a developer’s laptop included
• Switching backends changes OTEL_EXPORTER_OTLP_ENDPOINT; apps stay as they are

Harder:

• GlitchTip is company infrastructure: someone owns its upgrades, storage, backups, and capacity
• Instrumentation on Bun is young: check on the first app that ioredis, node:http, and runtime metrics report, and that pino records carry trace_id; keep ioredis and pino external to the server bundle
• Our four pieces are ours to maintain until Bun.otel and Start’s built-in tracing ship; traceSql follows the Bun.SQL call shapes Drizzle’s bun-sql driver uses, so a Drizzle or Bun upgrade runs its integration test first
• More spans cost more: a page that runs many queries exports a span for each. Lower the sampling ratio before dropping instrumentation
• @opentelemetry/* packages must move together, the SDK’s 0.x line still changes between releases, and the filter relies on the experimental onEnding hook — the guard test catches a release that breaks it
• Redaction is a rule to keep: a new log field or header capture can leak a token or personal data. Review logging changes like security changes
• GlitchTip lacks session replay and some newer Sentry features; SDK upgrades need a check against GlitchTip
• Dashboards and alerts depend on a backend this ADR doesn’t choose, so a product can’t assume any feature of it

Given up: Sentry’s replay and richer error UX, server-side error grouping in one tool, a single filter shared by every service on a host, and, while Bun lacks native tracing, maintained instrumentation for its HTTP server, Postgres client, and fetch. Accepted — apps stay vendor-neutral, and their privacy doesn’t depend on what receives their telemetry.