Serving LLMs in production: latency, cost and reliability

Once an AI feature has real users, the model call becomes a dependency like any other: it needs routing, limits, fallbacks and monitoring. A model gateway puts that in one place. The AI gateway note covers the wider control layer for agents, tools and governance.

ApplicationsProducts, internal tools, agents

Model gateway

• Access · authentication, per-team quotas and budgets
• Safety · input and output filters, PII handling
• Cache · repeated prompts and responses
• Routing · by task, cost and latency
• Resilience · timeouts, retries, fallback

• Managed model APIPrimary provider
• Managed model APIFallback provider or region
• Self-hosted modelsAutoscaled on queue depth, for residency or unit cost

Operations

• Tokens and cost per team
• Time to first token
• Tokens per second
• Error and fallback rates
• Quality evaluation
• Audit logs

Key design decisions

• Route by task: send simple tasks to smaller, cheaper models and reserve the largest models for work that needs them.
• Measure the right latency: track time to first token and tokens per second, not just total request time.
• Plan for failure: timeouts, retries with backoff and a fallback model or provider, so one outage does not take the feature down.
• Control spend: per-team budgets, caching of repeated prompts and alerts on cost per request.
• Build or buy: self-host only when data residency, unit cost at scale or customisation justify the operational load.