Production AI Service Architecture

Reference architecture. This describes an engineering pattern and current direction, not a specific production system built for an employer — see Experience for verified employment history and AI Systems for how this fits the overall positioning.

Most of what makes an AI-enabled feature reliable in production has nothing to do with model capability. It's the same discipline that makes any external-dependency integration reliable — applied to a dependency that is also non-deterministic.

Provider abstraction

Model providers change pricing, rate limits, and even output behavior over time, and a second provider is often needed for fallback or cost reasons. An AI service layer that sits between the application and the model provider — rather than calling the provider SDK directly from application code — is what makes swapping or adding a provider a configuration change instead of a rewrite.

Retrieval and tools as first-class dependencies

When a request needs external data (retrieval) or the ability to take an action (tool calling), those become dependencies of the AI service with their own latency, failure modes, and rate limits — the same as a payment provider or a third-party API. They deserve the same treatment: timeouts, retries with backoff, and circuit breakers, covered in Designing Reliable Third-Party Integrations.

Validate structured output, don't trust it

A model asked for JSON can still return malformed or semantically invalid JSON. Treating the model's response as untrusted input — validating it against a schema before it reaches downstream code — is the same discipline applied to any external API response, just with a higher failure rate to plan for.

Async by default

Model calls are slow relative to a typical API request. Where the product allows it, moving AI work off the synchronous request path — into a queue, with a status the client can poll or subscribe to — avoids tying user-facing latency directly to model provider latency. This is the same tradeoff explored in Designing Async Processing with Kafka and Worker Queues, applied to a non-deterministic dependency instead of a deterministic one.

Cache what's safe to cache

Identical prompts against a deterministic-enough configuration are a legitimate caching target — both for cost and latency. The cache key has to account for everything that affects the output (prompt, model version, relevant parameters), and the cache should be treated as an optimization, not a correctness mechanism.

Observability has to cover cost and quality, not just uptime

A model call can be "successful" — a 200 response — and still be wrong, too expensive, or slower than acceptable. Observability for an AI service needs token usage and cost per request, latency distributions, and some signal on output quality (even a coarse one), alongside the standard error-rate and uptime metrics covered in What Makes a Production System Observable?

Fallback behavior is a product decision, not just an engineering one

When the primary model provider is degraded or a call fails validation, the system needs a defined fallback — a secondary provider, a cached response, a degraded-but-safe default, or an explicit "try again" to the user. Deciding this ahead of time, rather than during an incident, is what keeps an AI-enabled feature from becoming a single point of failure for the product around it.