ADR-0025 AI SDK

Context

Products will add AI features: a chat assistant, summaries, extraction from documents, actions the model takes through tools. The surrounding decisions are made, and each puts a rule on this one:

• Framework: TanStack Start with server functions and streaming SSR (ADR-0002); server routes at /api/* for anything consumed as an HTTP API (ADR-0036)
• Validation: Zod 4 at every boundary (ADR-0005). A tool’s arguments and a model’s structured answer are boundaries too — the model is a caller we don’t control
• Auth: Better Auth sessions, checked on the server (ADR-0009)
• Rate limiting: one limiter in Start’s global request middleware, keyed by user ID (ADR-0010). Its starting write limit is 300 requests per user per minute — sized for forms, not for calls that cost money
• Client state: server data in Query, UI state in the URL and React (ADR-0012)
• Observability: OpenTelemetry, filtered in the app before export; telemetry already carries user IDs, so what we record is a privacy decision (ADR-0015)
• Tests: no real third-party calls (ADR-0014) — a model is one
• Rendering: model text is Markdown (ADR-0021), with highlighted code (ADR-0022)

What is open: the SDK that talks to models, and the rules for using it. The model provider is not open here. Which provider, model, and region is a per-product call, because it depends on the data the product sends and on the contract terms it can get. It goes in the product’s own ADR (How we decide, rule 5), and the Decision section says what that ADR must cover.

Three facts shape the answer:

• A model call is a paid request to a third party. It costs money per token, it can run for a minute, and whatever we put in the prompt leaves our infrastructure. Rate limits (ADR-0010) and data protection (GDPR; we are an EU company) apply from the first call
• Model output is untrusted input. Anything the model reads — a user message, a document, a tool result, a web page — can steer what it writes and which tools it calls. The OWASP Top 10 for LLM Applications (2025) lists prompt injection, improper output handling, excessive agency, and unbounded consumption. We treat the model’s text, tool arguments, and structured answers like a webhook payload (ADR-0005)
• The SDK is pre-1.0 and moves fast. @tanstack/ai is at 0.70.0. The question is how to adopt it safely, not whether it is finished

Requirements:

• Streaming chat in React with stop and error states, fitting Start’s server functions and API routes
• Provider-agnostic: swapping or adding a provider changes one adapter in one place on the server
• Typed tools whose schemas are Zod, and validated structured output
• Runs on Bun in our container, with no hosted gateway or platform in the path (ADR-0003)
• OpenTelemetry spans and token metrics, with prompt content off by default
• A scripted fake model, so tests need no network and no key

Options considered

SDK

A. TanStack AI

@tanstack/ai holds the core: chat(), tools, structured output, middleware, and server helpers. @tanstack/ai-react holds the hooks. Each provider is an adapter package: OpenAI (@tanstack/ai-openai 0.30.1), Anthropic (@tanstack/ai-anthropic 0.22.1), Gemini, Vertex, Bedrock, Mistral, Ollama, OpenRouter, Groq, Grok, and more — plus openaiCompatible for any OpenAI-compatible endpoint.

Pros

• Same maintainers and organisation as Router, Query, Start, and Form; MIT. @tanstack/ai 0.70.0 (2026-10-19) and @tanstack/ai-react 0.31.0. About 594 k downloads a week (2026-10-12 to 2026-10-18)
• chat() returns an async iterable of events; toServerSentEventsResponse() turns it into an SSE Response. In React, useChat() reads it through fetchServerSentEvents() for an HTTP route, or through a fetcher for a Start server function. stop() aborts the request, and an AbortController passed to chat() stops the model call too
• Tools are defined once with toolDefinition() and a Zod schema, then implemented with .server() or .client(). The SDK validates each call’s arguments before the tool runs; a failure goes back to the model as a tool error. needsApproval pauses a tool for user approval
• Structured output through outputSchema (Zod 4.2+, ArkType, or Valibot)
• Server-built context is passed to tools and middleware, and is never sent to the model — a ready seam for the signed-in user and injected dependencies
• An OpenTelemetry middleware (@tanstack/ai/middlewares/otel) emits a span per chat() call, per model turn, and per tool call, with GenAI attributes and token and duration histograms. Prompt and completion capture is off unless captureContent: true — we read the default in the 0.70.0 source
• fakeText() from @tanstack/ai/testing is a scripted adapter: queued answers, tool calls, errors, estimated token usage
• When the next call goes to a different provider, the SDK rewrites saved history (thinking signatures, tool-call IDs) to fit — which makes a fallback provider realistic
• A devtools panel for messages and tool calls (@tanstack/react-ai-devtools). Pure library: no platform layer, any host that returns a Web Response

Cons

• Pre-1.0 and churning. 102 releases in about 10½ months; 25 new minor lines between 0.43 (2026-08-04) and 0.70 (2026-10-19). Breaking changes have shipped inside 0.x — the HTTP wire format, sampling options moved into modelOptions, TextPart’s props, the MCP SDK packages — each with a migration guide
• Every adapter and @tanstack/ai-react peer-depends on @tanstack/ai ^0.70.0, which for a 0.x version means the same minor only
• Younger and smaller than the AI SDK: about 594 k downloads a week against about 18.4 M, and fewer answers online. Its docs on GitHub follow main
• Its docs show patterns we forbid by default: a key held in the browser (BYOK), a provider chosen by the client (forwardedProps), and a server function whose validator is only a TypeScript cast
• Provider SDKs ride along as dependencies of the adapters, with ranges we don’t control
• @tanstack/ai-react 0.31.0 has @tanstack/markdown ^0.0.13 as a hard dependency (not optional), used by its prebuilt TextPart. Its Markdown renderer is a second, older copy of the library behind ADR-0021

B. Vercel AI SDK

ai 7.0.148 (Apache-2.0, published 2026-10-21), @ai-sdk/react 4.0.152, and provider packages such as @ai-sdk/openai 4.0.99.

Pros

• A stable major, and the largest user base: about 18.4 M downloads a week for ai, 5.7 M for @ai-sdk/react
• Dozens of first-party provider packages; useChat; tool calling; a mock model in ai/test (MockLanguageModelV4); OpenTelemetry through @ai-sdk/otel 1.0.148
• A ready agent abstraction, ToolLoopAgent

Cons

• Once its OpenTelemetry integration is registered, inputs and outputs are recorded by default; privacy takes recordInputs: false and recordOutputs: false. The safe setting is the opt-out — the opposite of ours
• Stewarded by Vercel, the platform company we declined as a deploy target (ADR-0003) and which now owns Better Auth (ADR-0009); its docs list Vercel AI Gateway first among providers. The licence and running it in our own container keep the exit open, and nothing in the SDK requires the platform
• A second release train next to TanStack’s, and 236 ai releases on npm in the last 90 days: stable semver, but constant dependency noise

C. Provider SDKs directly

openai 7.39.0 (Apache-2.0) and @anthropic-ai/sdk 0.137.0 (MIT).

Pros: nothing between us and the provider; first access to new provider features; the smallest dependency set; about 25.3 M and 28.3 M downloads a week. Cons: each has its own message, tool, and stream shapes, which would leak into our features — a provider switch becomes a rewrite. No React hook, no tool loop, no cross-provider fake: every product would rebuild SSE transport, abort, usage, tool validation, and OTel spans. @anthropic-ai/sdk is itself pre-1.0. (The adapters in A wrap these same SDKs.)

D. LangChain.js

langchain 1.5.18, @langchain/core 1.2.20, and @langchain/langgraph 1.5.1, all MIT.

Pros: stable 1.x; wide integrations; LangGraph for stateful, multi-step workflows; about 1.7 M downloads a week for langchain. Cons: a layered agent framework — a create_agent harness, LangGraph orchestration, LangSmith tracing — where we need a thin SDK. Its abstractions would sit between our code and the model, and langchain depends on both LangGraph and the langsmith client.

E. Mastra

@mastra/core 1.78.0 and @mastra/react 1.11.0, Apache-2.0 apart from code under ee/ directories, which has its own licence.

Pros: stable 1.x; agents, workflows, and memory in one TypeScript framework; about 1.3 M downloads a week for @mastra/core. Cons: a framework that wants to own the agent runtime — @mastra/core has 30 dependencies, hono and ws among them. We would adopt its runtime, not an SDK, and check the ee/ licence feature by feature.

Where the model call lives

• Start server function: createServerFn({ method: "POST" }) returning toServerSentEventsResponse(...), called from useChat({ fetcher }). For our own UI, as ADR-0036’s convention says. Same path as every write: limiter, CSRF check, validation
• Server route under /api/*, read with fetchServerSentEvents(): for consumers outside our React app — mobile, partners, scripts. The limiter covers it; the server-function CSRF check does not, so it must not accept the session cookie

Decision

TanStack AI is the company-standard AI SDK for all new apps: @tanstack/ai and one provider adapter on the server, @tanstack/ai-react in the browser. It is accepted while pre-1.0, pinned to exact versions. This ADR picks the SDK and the rules for using it — not a model provider. Each product chooses its provider in its own ADR.

Concretely:

• Pre-1.0, accepted and pinned — as ADR-0008 did for Drizzle v1. No ^ or ~ on any @tanstack/ai* package or on @tanstack/react-ai-devtools. Today: @tanstack/ai 0.70.0, @tanstack/ai-react 0.31.0, and the adapters a product’s ADR names (for example @tanstack/ai-openai 0.30.1 or @tanstack/ai-anthropic 0.22.1). They move together, in one Renovate group (ADR-0016), because each adapter and @tanstack/ai-react peer-depends on the same @tanstack/ai minor. Transitive @tanstack/ai-* packages (ai-client, ai-utils, ai-event-client) are held by the lockfile, which CI installs frozen. Upgrades are deliberate: read the release notes and migration guides, run the tests on the fake model, and deploy server and browser bundle together. Revisit when 1.0 ships, in a new ADR
• Model calls run only on the server. chat() and the adapters are imported only by server code; the browser imports @tanstack/ai-react and never an adapter package. The provider key is a server environment variable, validated at boot with Zod (ADR-0005) and never exposed to the client (no VITE_ prefix). No BYOK (@tanstack/ai-react/byok): a key held in the browser changes who the provider’s customer is, so a product that wants it writes its own ADR
• One adapter boundary. Exactly one module, src/server/ai/adapter.ts, imports a provider package and reads its key. It returns the SDK’s adapter type (AnyTextAdapter) built from validated config. Feature code, server functions, and routes receive the adapter as a parameter and never import a provider package; tests pass fakeText() in its place, and a product adding a fallback provider adds a second factory here and nothing else. The provider and model come from config, never from the request — the SDK’s runtime-switching example reads forwardedProps.provider from the body, which would let a client pick a provider no ADR approved. A product that offers a model choice maps a Zod enum to an allowlist on the server
• Thin handlers, rules in plain modules. A server function or API route handler only parses, authorizes, limits, and calls a feature function. One module, src/server/ai/chat.ts, is the only one that calls chat(): it takes the adapter, the validated messages, the user, and an abort signal as parameters, and attaches the middleware (OpenTelemetry, usage, tool budget, error mapping). Prompts and the tool set live in plain modules that take their dependencies — adapter, clock, data access through context — as parameters (ADR-0014). The system prompt is set on the server with systemPrompts; user text never becomes a system message
• Server function for our UI, a server route for outside clients (ADR-0036). Our UI calls a POST server function — createServerFn({ method: "POST" }).validator(schema) — that returns toServerSentEventsResponse(chat(...)), read by useChat({ fetcher }). Never GET: the limiter skips reads (ADR-0012). A server route under /api/v1/ai/* is for non-browser consumers only. It parses with chatParamsFromRequestBody(await request.json()) inside try/catch, answering 400 itself, and rejects any body that is not application/json. It authenticates through apiCaller, with a token and never the session cookie, so it needs no CSRF check — ADR-0013’s covers server functions only. (Amended 2026-11-13: this ADR first used a Hono route with Hono’s csrf(); ADR-0036 replaced Hono with Start’s server routes.)
• Authenticated first, then limited per user. Every AI entry point resolves the caller through Better Auth before it touches the SDK — the session for the server function, the token for the API route — and answers 401 otherwise; there is no anonymous AI endpoint. ADR-0010’s generic write limit is too loose for model calls, so AI routes also consume a stricter ai scope from the same limiter, keyed by user ID — starting at 10 requests per user per 60 s, tuned per product. One request is a whole chat() run, which can call the model several times. Past the limit: 429 with Retry-After, as in ADR-0010
• Cap what one request can cost. The limiter counts requests, not tokens, so each run is also bounded:
  • Zod limits on message count and text length at the boundary
  • an explicit agentLoopStrategy such as maxIterations(n) — the default is five model turns, and one turn can issue several tool calls — plus a tool-call budget in middleware (onBeforeToolCall)
  • an output-token cap in the adapter’s provider-native modelOptions
  • a per-user daily token budget, counted from onUsage in the same Valkey (ADR-0010); the number is set in the product’s ADR
  • one AbortController passed to both chat() and toServerSentEventsResponse(), so Stop or a closed tab stops the model call too
  • a run deadline below the edge’s write timeout: Traefik’s respondingTimeouts.writeTimeout “covers the time from the end of the request header read to the end of the response write”, and ADR-0017 starts it at 60 s, so a longer stream is cut at the edge. Abort runs at 50 s (starting value); a product that needs longer runs changes the edge value deliberately
• Zod at every boundary the SDK opens:
  • Request: chatParamsFromRequest checks only the AG-UI request shape. We validate with our own Zod schema, drop client-sent system and developer messages, and never pass the client’s tools to chat() — the server builds the tool list. forwardedProps are client-controlled: allowlist every field before use
  • Tools: toolDefinition() with a Zod inputSchema and outputSchema, bounded with .max() and enums. Not raw JSON Schema, which types args as unknown. The SDK validates each call; the handler still authorizes
  • Structured output: outputSchema with Zod 4.2+. The non-streaming path validates; on the streaming path (stream: true) the SDK leaves validation to us — parse with the same schema (parseWithStandardSchema) before using the object
• Tools run on the server unless they only touch the UI. .server() for data, secrets, and side effects, taking dependencies and the signed-in user from chat({ context }) — built on the server, never from the body. .client() only for UI effects such as a toast or navigation: it runs in the browser and its result comes back from the client, so it is untrusted input. Handlers authorize by the session user, not by the conversation: the model is not an authority, and an approval is a UX step, because approvals resume from the message history the browser sends back. A tool’s result streams to the browser too (TOOL_CALL_RESULT), so it returns only what the signed-in user may see. Least privilege: the smallest tool set per route, read-only by default; writes, sends, deletes, and spending use needsApproval: true and re-check permission in the handler
• State from the hooks, text through our Markdown. The browser imports useChat and the other hooks from the @tanstack/ai-react root entry only — never @tanstack/ai-react/ui (the prebuilt Chat, ChatMessage, and TextPart), and never TextPart or any other built-in Markdown renderer from the SDK. TextPart renders with its own copy of @tanstack/markdown, an exact pin on the 0.0.x line, which would put a second Markdown renderer with its own settings next to ours (see Consequences). The hooks own state and streaming; each text part is rendered by the shared Markdown component from ADR-0021 (src/components/markdown/markdown.tsx, configured in src/lib/markdown.ts). Chat does not get its own settings: allowHtml is never set, remote images are dropped unless their origin is on ADR-0021’s allowlist, and the streaming extension is always on
• Model output is untrusted input. Text parts render only through ADR-0021’s safe Markdown path — never raw HTML, never dangerouslySetInnerHTML — and code blocks through ADR-0022. ADR-0021’s rules for links and images apply, because a model can be steered into emitting a link or image whose URL carries conversation data out. Structured answers are parsed by Zod before use. Model text never becomes a URL, file path, SQL, or shell argument without validation
• No personal data in logs or traces by default.
  • Wire otelMiddleware({ tracer, meter }) from @tanstack/ai/middlewares/otel once, in chat.ts. ADR-0015 already installs @opentelemetry/api; the SDK’s peer range is >=1.9.0
  • Leave captureContent at false, its default: spans then carry model, token counts, finish reason, and tool names and outcomes — never prompts, completions, or tool arguments and results
  • Never set debug: true outside development (it prints raw provider chunks and tool calls), and never write prompts, completions, or tool input and output to pino logs. Log user IDs, never emails
  • A product that must capture content for debugging or evaluation says so in its provider ADR, with a redact function and a retention shorter than the backend keeps traces. If redact throws, the SDK records [redaction_failed] instead of the content
  • Errors leak too. The SDK puts the exception message, and for some adapters the provider’s error body, on the RUN_ERROR chunk that streams to the browser, and it records tool exceptions on spans even with capture off. Tool handlers throw generic errors with no personal data, and the server replaces the RUN_ERROR message with a generic one plus the request ID before the stream leaves (ADR-0013, ADR-0015)
• Chat state is client state. useChat owns the live conversation — messages, streaming, stop, errors. It is neither server data nor URL state (ADR-0012): don’t copy messages into Query or Zustand. Saved conversations, if a product keeps them, are server state (Query with GET server functions, writes by POST) and a product decision with its own retention and deletion rules
• Tests use a fake model, never a real one. Server and integration tests inject fakeText() through the adapter boundary — scripted answers, tool calls, and errors, with state.callCount to assert on. Component tests give useChat a stream() connection over chat() with a fake adapter. No real model call in CI; MSW’s onUnhandledFrame: "error" (ADR-0014) catches a leak. Every app with AI features adds guard tests next to ADR-0014’s four: a signed-out request is refused and the fake is never called; a request past the limit gets 429 with Retry-After; an oversize body is rejected before the model; a client system message and a client tools list never reach the adapter. Answer-quality evals belong to the product
• Devtools in development only: @tanstack/react-ai-devtools and @tanstack/react-devtools as devDependencies. The panel shows messages and tool inputs and outputs; it doesn’t ship
• Out of scope, each needing its own ADR: the model provider; conversation persistence (@tanstack/ai-persistence), memory, and resumable streams; MCP; code mode and sandboxes; media generation and realtime voice; embeddings and retrieval

What each product’s provider ADR must cover:

• Provider, model, and region. Exact model IDs, not floating aliases. Where prompts are processed and stored — data residency: an EU region or endpoint, or self-hosting through @tanstack/ai-ollama or openaiCompatible. Transfers outside the EEA need a GDPR Chapter V mechanism. A gateway (OpenRouter, Vercel AI Gateway) puts one more processor in the path
• Personal data in prompts. Which categories reach the provider; what we strip or pseudonymise first (data minimisation, GDPR Article 5(1)(c)); the lawful basis; a data processing agreement with the provider and its sub-processors (Article 28)
• Retention and training terms. How long the provider keeps prompts and outputs, abuse-monitoring copies included; whether inputs train models; zero-retention options; and where this is written — the contract, not a marketing page
• Cost. Price per token, expected usage per user, a monthly budget with an alert, and the per-user request and daily token limits from this ADR
• Fallback. What happens when the provider is down, throttles, or retires the model: a second provider behind the adapter boundary, or the feature degrades and says so. Timeouts and retries — a retry is a second charge
• Users. The UI says the user is talking to an AI system (EU AI Act Article 50(1), applicable since 2026-08-02); what is stored, and how a user deletes it
• Reach. The tools and data the model can touch

The deciding factors: the same maintainers and release train as Router, Query, Start, and Form; Zod schemas that type tools and answers and check every call; content capture off by default in its OpenTelemetry middleware; a scripted fake model for offline tests; and no platform layer between us and the provider. We accept a 0.x API that changes every few days, in exchange for exact pins and upgrades on our schedule — the same bargain as Drizzle v1 in ADR-0008.

Consequences

Easier:

• A provider is one adapter at one boundary: changing, adding, or failing over a provider touches one module, and the SDK rewrites saved history when the next call goes to a different provider
• One schema language: Zod types a tool’s arguments and results and a structured answer, and the SDK checks each call
• Every AI request takes the same path as every write: useChat → POST server function → request ID → rate limiter → CSRF check → session → Zod validation → feature module → adapter → provider
• Spans for each run, model turn, and tool call — with token counts, and cost where the provider reports it — appear with the rest of the app’s telemetry (ADR-0015) without a word of prompt
• Tests run offline and give the same answer every time

Harder:

• Pre-1.0 risk, plainly. Any minor release can break types or behavior: the wire format, option names, and component props have all changed inside 0.x. Exact pins and a grouped Renovate pull request make each upgrade ours to schedule, but not free — every one needs release notes, a test run, and a joint deploy. If breaking changes keep landing at this rate after 1.0, this decision is wrong and a new ADR replaces it
• One core minor bump forces matching bumps in every adapter and @tanstack/ai-react: a fix in the core can’t be taken alone
• A second copy of @tanstack/markdown still installs. @tanstack/ai-react 0.31.0 lists it under dependencies as ^0.0.13, and a caret on 0.0.x allows only that exact patch, so the package manager keeps it next to our 1.x (ADR-0021). We never import it: the hooks in the root entry don’t load it, only @tanstack/ai-react/ui does (checked in the 0.31.0 package), and the rule above keeps that entry out of our code. The copy still counts for the lockfile and dependency audits, and keeping /ui out is a review rule. Check with bun why @tanstack/markdown: two versions are expected, and a production bundle that contains the old one means /ui or TextPart got imported. Revisit when @tanstack/ai-react moves to @tanstack/markdown 1.x
• Server and browser are coupled by the stream’s wire format. After an upgrade that changes it, tabs opened before the deploy can fail until reloaded, and rolling deploys run two versions briefly (ADR-0010). Show errors with the request ID and a reload hint
• The SDK doesn’t authenticate, rate limit, redact, or sanitize — those rules are ours, enforced by review and the guard tests. Review AI changes like security changes
• Cost control is ours: the limiter counts requests, not tokens, so the loop, token, tool-call, and deadline caps live in our code
• Docs and examples need translating. The docs on GitHub follow main; the published package ships src/ and agent skills/, not docs/. Its server-function sample calls .inputValidator(...), which Start 1.170’s createServerFn types don’t have (ours is .validator()), and validates nothing at runtime
• Adapters carry the provider’s SDK with their own range: @tanstack/ai-anthropic 0.22.1 depends on @anthropic-ai/sdk ^0.97.1 (2026-05-19) while npm’s latest is 0.137.0. A provider-SDK fix can wait for an adapter release
• OpenTelemetry’s GenAI conventions are still in Development status, and the middleware emits gen_ai.system where the current convention says gen_ai.provider.name. Build dashboards on what we see in Tempo, and recheck on every upgrade
• A smaller community than the AI SDK, and fewer first-party providers; gaps go through OpenRouter or openaiCompatible, at the price of a gateway in the data path

Given up: the AI SDK’s stable v7 and larger ecosystem, and the minimalism of calling a provider SDK directly. Accepted — a typed, provider-agnostic SDK from our own TanStack stack, with prompt capture off by default, matters more than maturity for a company default; products with different needs write their own ADR. Reopen this decision when 1.0 ships, if the project stalls, or if a product needs a provider this SDK cannot reach. Revisit the Markdown rule when @tanstack/ai-react moves to @tanstack/markdown 1.x: only then can its renderer share one copy and one policy with ours.