FastAPI Architecture for Production AI Applications

A route should translate an authenticated HTTP request into an application command, then translate an application result into a documented response. It should not own provider SDK parsing, retry loops, state transitions, database details, or queue recovery. Otherwise every endpoint becomes a different orchestration system.

The architecture in this guide separates routes, request validation, authorization, idempotency, job records, a queue/worker lane, a provider adapter, output validation, persistence, and observability. Each boundary can fail independently and should expose a stable failure category.

Production-oriented means the design addresses production concerns. The synthetic fixture cannot prove throughput, reliability, security completeness, crash recovery, multi-worker consistency, cloud scale, or an SLA.

CreateAnalysisRequest defines what a caller may send. AnalysisAcceptedResponse defines the 202 acknowledgement. AnalysisResultResponse defines the polling representation. ErrorEnvelope defines failures without leaking internal exceptions. Version these public contracts independently from provider SDK objects.

FastAPI request bodies use Pydantic models, while response models can validate and filter returned data. Configure extra-field and coercion behavior deliberately. Transport validation checks shape and types; business validation checks supported analysis types and tenant rules; provider validation checks untrusted external output.

Provider-specific objects must stop at the adapter. A client should not break because a provider renames a field, changes an error class, or adds metadata.

The fictional API accepts a document ID, analysis type, idempotency key, metadata, and tenant-scoped principal. A new valid request creates an analysis record, moves it to queued, and returns the identity conceptually. An in-memory worker later claims the job, calls a fake provider, validates the result, and stores a terminal state.

Nine POST attempts cover four accepted jobs, an exact replay, a conflicting reuse of a key, invalid input, an unsupported operation, and a policy block. The queued jobs cover success, a single timeout followed by success, invalid provider output routed to review, and another success.

Everything is synthetic. The runner uses Python 3.12 standard library, fixture data, and memory only. It starts no FastAPI server, performs no network call, and writes no database.

A synchronous HTTP completion can fit short, bounded work with predictable dependencies and a response deadline the client can tolerate. The route still needs cancellation, timeout, provider normalization, and output validation.

A job model is safer when work is long-running, involves multiple provider/tool steps, needs retries, processes large documents, must survive client disconnects, or requires worker isolation. POST accepts work and returns 202 plus an analysis ID; GET exposes progress and the terminal result.

Writing async def allows cooperative concurrency for awaitable I/O. It does not create durable background work, make CPU-heavy code non-blocking, or preserve a task after process failure. Streaming can improve interactive delivery but is not a job queue and does not supply recovery.

The production-oriented flow is validate, authorize, resolve idempotency, create the job record, enqueue its identity, and return 202 with analysis_id and status_url. A database transaction plus outbox or another atomic handoff pattern is often needed so the record and queue message cannot diverge.

The worker claims by analysis ID, validates the current state, records an attempt, calls the provider adapter, validates output, and commits a terminal state. The client polls a tenant-scoped GET endpoint rather than holding the original POST open indefinitely.

The fixture uses a deque and dictionaries. This demonstrates sequencing only; it does not prove atomic enqueueing, durable delivery, crash recovery, leases, or multi-worker behavior.

FastAPI documents BackgroundTasks for functions that run after a response is returned. It can be appropriate for small same-process follow-up work. The documentation itself points to larger queue tools when heavy computation should run in other processes or servers.

BackgroundTasks does not automatically persist a job, provide distributed claiming, survive process termination, enforce retry policy, or coordinate multiple workers. Long-running AI work usually needs an external queue/worker boundary plus persistent operation state.

Do not choose a queue product before defining delivery, ordering, retry, idempotency, timeout, cancellation, and recovery requirements.

An AIProvider interface can expose analyze(document, analysis_type, deadline) and return a provider-neutral candidate result. The adapter owns SDK configuration, request translation, timeout mapping, rate-limit mapping, response parsing, and safe diagnostic context.

Normalize success, timeout, rate-limited, transient error, permanent error, and invalid output into application categories. Do not return raw exceptions, headers, request bodies, secrets, or provider-specific stack traces to API clients.

The fake provider reads authored outcomes from the fixture. It proves adapter-shaped control flow, not any real provider's reliability or structured-output behavior.

Validate exact required fields, types, enums, evidence references, lengths, and format rules before committing a result. Provider JSON or structured-output modes can reduce syntax failures but do not prove that the answer is factually correct, grounded, safe, or appropriate.

The simulator requires exactly analysis_type, summary, labels, and evidence_ids; it checks types, the requested analysis type, non-empty summary and evidence. A missing evidence list routes the operation to review_required with provider_invalid_output.

Use review_required for ambiguous provider output, policy-sensitive analysis, conflicting evidence, exhausted retries that require adjudication, or an invalid final result where automatic failure would discard useful evidence. Define which cases are eligible and who may resolve them.

A review record should contain safe input references, normalized candidate output, validation failures, attempts, policy version, and the requested decision. It should not request hidden chain-of-thought or expose unrelated tenant data. The fixture routes one invalid result to review without assigning a confidence score or performing human judgment.

Use the existing evaluation pipeline guide for dataset design, graders, slice analysis, regressions, and release gates. This Article owns the HTTP and execution architecture around that quality system.

A client timeout bounds how long the caller waits. A request handler deadline protects the HTTP tier. A provider-attempt timeout bounds one external call. A worker timeout bounds one execution attempt, while an overall job deadline limits the entire operation across retries.

Retry only failures likely to be transient: selected timeouts, eligible server errors, or rate limits when server guidance and policy permit. Invalid input, unsupported operations, policy blocks, state conflicts, and invalid provider schemas should not loop.

Use capped exponential backoff with jitter conceptually and honor applicable Retry-After guidance. The fixture does not sleep: it retries one authored timeout once so execution stays deterministic. No latency result is measured.

Scope an idempotency key to the authenticated tenant or project. Store it with a canonical request hash and the created analysis ID. A repeated key with the same hash can return the existing operation; the same key with a different hash should return a conflict.

The request hash should cover the semantic command, not volatile transport fields. Protect the key and hash with a uniqueness constraint, and resolve races transactionally. Retention and reuse policy must be documented.

Idempotency does not guarantee exactly-once execution. Workers and downstream effects still need idempotent writes, claim/lease logic, and reconciliation because messages or attempts can be repeated.

The fixture uses accepted, queued, running, succeeded, failed, and review_required. Succeeded, failed, and review_required are terminal for this teaching model. Production cancellation can be added only with explicit rules for who can request it and what happens if work is already running.

Allow transitions rather than merely listing statuses: accepted to queued, queued to running, and running to a terminal outcome. Use conditional updates or versions so two workers cannot both overwrite state without detection.

Store attempt count and failure code separately from status. A failed operation with provider_timeout needs different handling from a review_required operation with invalid output.

A durable analysis record commonly contains analysis_id, tenant/project identity, idempotency key, request hash, status, analysis type, provider configuration reference, attempt count, timestamps, result reference, and failure code. Sensitive source documents should have separate access and retention rules.

Commit state transitions atomically, use optimistic concurrency or guarded updates, and make result writes idempotent. Decide how queued records are reconciled if message delivery fails and how running records are recovered after an expired lease.

The simulator stores everything in dictionaries and a deque. It proves none of durability, restart recovery, transaction isolation, multi-worker consistency, backups, or disaster recovery.

A useful envelope contains error.code, a safe message, request_id, retryable, and optional sanitized details. Correlate the public request ID with internal telemetry without exposing raw provider exceptions or confidential content.

Map validation failures to 400 or framework-appropriate validation semantics, authentication/authorization to 401/403, missing tenant-scoped operations to 404, idempotency or state conflicts to 409, rate limiting to 429, and internal/provider-boundary failures to suitable 5xx responses.

Retryable is an application promise, not a guess. It should reflect method semantics, idempotency protection, the failure class, remaining deadline, and policy.

Authentication establishes identity. Authorization checks tenant/project membership, resource ownership, allowed analysis types, document access, and operation visibility. A valid API key alone does not prove permission for every document or action.

Resolve a principal through a dependency, then pass a constrained auth context into the application service. Every create and read path must enforce the same tenant boundary; avoid direct lookups that reveal whether another tenant's analysis exists.

The fixture supplies authored principal permissions and one policy block. It does not implement or prove an authentication system.

/health should answer whether the process and event loop can respond. /readiness should answer whether required configuration and dependencies are ready enough for traffic—for example persistence, queue access, schema/migrations, and provider configuration.

Readiness should normally inspect configuration or lightweight dependency state, not perform a paid or mutating provider inference on every probe. Deployment platforms can remove an unready instance from traffic while leaving a live process running for diagnosis.

The fixture evaluates two authored readiness states and records one failure when the queue is unavailable. It does not start a server or inspect real dependencies.

FastAPI Depends can supply the provider adapter, repository, queue, and principal to thin routes. Tests can replace those dependencies with deterministic fakes without importing provider SDK behavior into domain logic.

Current FastAPI guidance recommends the lifespan parameter with an async context manager for startup and shutdown work. Initialize reusable database pools, HTTP clients, queue clients, or model resources before serving, then close them after the application stops.

Do not create expensive reusable clients per request unless their library requires it. Keep resource ownership explicit so timeouts, connection limits, TLS, cleanup, and tests have one configuration boundary.

Carry request_id and analysis_id across the route, repository, queue message, worker, provider attempt, validation, and result update. Record route, tenant-safe reference, state transition, attempt, provider alias, duration from real telemetry, failure code, queue delay, and terminal status.

Metrics aggregate counts and distributions; logs capture categorized events; traces connect the path across components. OpenTelemetry defines these as distinct signals that can be correlated.

Do not log secrets, raw confidential documents, full provider payloads by default, or hidden chain-of-thought. Apply redaction, access, retention, and sampling policy before adding content to telemetry.

Client failures include invalid_request and unsupported_operation. Policy includes policy_block. Idempotency and state conflicts identify deterministic application races or misuse. Provider failures include timeout, rate limited, provider error, and invalid output. Queue and persistence failures remain separate.

Normalize internal details into stable public codes while retaining sanitized operator evidence. A provider timeout must not look like client validation, and an unavailable queue must not be reported as successful acceptance.

Review taxonomy over time, but version changes so dashboards, alerts, and clients do not silently reinterpret old failures.

An async route can interleave awaitable I/O, but CPU-heavy parsing, embeddings, local inference, or image processing can block the event loop. Isolate CPU-heavy work in worker processes, process pools, or an external job system appropriate to its resource and cancellation model.

A production deployment may place a reverse proxy or load balancer before stateless API replicas, with external persistence and queue services shared by workers. Multiple server workers consume separate memory, so in-process queues and caches cannot provide cross-worker consistency.

Add health/readiness probes, environment-managed secrets, resource limits, graceful shutdown, migration strategy, rollback, and measured capacity planning. No particular cloud or worker count is universally best.

The two artifacts reproduce request validation, policy checks, idempotent replay and conflict, state transitions, an in-memory queue, provider normalization, one bounded retry, invalid-output review, readiness evaluation, and terminal status retrieval.

Raw-byte matching proves the deterministic authored result in the tested Python environment. It does not run FastAPI, Pydantic, a provider, a database, a network, or a durable queue.

Implement durable transactional state, an atomic enqueue/outbox boundary, worker claims and leases, idempotent terminal writes, bounded deadlines, cancellation rules, queue reconciliation, migrations, backups, and disaster-recovery tests. Exercise process termination at each transition.

Add contract and integration tests, provider-adapter fixtures, authentication/authorization tests, tenant-isolation tests, load and capacity tests using representative authorized data, observability alerts, privacy controls, secrets rotation, incident runbooks, and deployment rollback.

Keep each neighboring concern in its own layer: agent orchestration controls the decision loop, RAG controls evidence retrieval, evaluation controls quality gates, and this FastAPI design controls HTTP acceptance and backend execution.