AI gateway: one control point for models, agents and tools

As AI use spreads across teams, each application tends to call models directly, with its own keys, limits and logging. An AI gateway replaces that with a single control point between consumers and every model, agent and tool, so access, safety, cost and audit are handled once and applied consistently.

ConsumersApplications, agents, internal tools, developer assistants

AI gateway

• Identity · workload and user identity, no shared provider keys
• Policy · which teams may use which models, tools and data classes
• Guardrails · prompt injection, PII redaction, output checks
• Routing · by task, cost, latency and data residency
• Resilience · timeouts, retries, fallback across providers
• Cost · quotas, budgets, caching, chargeback per team
• Tool access · agent calls to MCP servers and APIs, with scoped permissions

• Managed model APIsMultiple providers and regions
• Self-hosted modelsFor residency, unit cost or customisation
• Tools and dataMCP servers, enterprise APIs, retrieval

Observability and audit

• Request and token logs
• Cost per team and use case
• Latency and error rates
• Guardrail events
• Quality evaluation
• Audit trail

Through the gatewayConsistent, measured, auditable

1. RequestApp or agent, with its identity
2. AuthenticateIdentity and entitlements
3. Policy and budgetModel, data class, quota
4. Input guardrailsInjection, PII
5. RouteModel, provider or tool
6. Output guardrailsSafety, data leakage, schema
7. Log and meterTokens, cost, audit

Provider keys and tool secrets live only in the gateway, never in applications

Key design decisions

• Centralise credentials: provider keys and tool secrets stay in the gateway. Applications and agents authenticate with their own workload identity.
• Policy as code: model, tool and data-classification rules are versioned, reviewed and tested like any other code.
• Govern tools, not only models: agents act through tools, so tool calls (for example over MCP) go through the same gateway, with least-privilege scopes, approval for high-impact actions and a full audit trail.
• Provider independence: one API in front of several providers lets models be added, switched or retired without changing applications.
• Stay fast on the hot path: keep gateway overhead to milliseconds, stream responses, and move heavy checks to asynchronous review where the risk allows.
• Evidence for compliance: the gateway log records who used which model or tool, with what data and what result, which is the record AI governance frameworks such as the EU AI Act expect.
• Build or buy: managed gateways and API management platforms cover most needs. Build only the policy and integrations specific to the organisation.