Before approving an AI gateway for production, draw the systems it can reach. The model API is usually only one of them.
The same service may hold provider keys, inspect prompts, load custom guardrail code, route to internal APIs, connect to MCP servers, and inherit a cloud workload identity. Its operational importance follows that authority.
The finding that changes the review
The LiteLLM compromise chain connected several weaknesses across authentication, guardrail code, routing, and credentials. Different paths had different prerequisites. The combined lesson is that gateway features can span several security domains inside one process.
Treating that process as an ordinary stateless proxy leaves important permissions out of the architecture review.
Review each connection as a separate grant of authority. A gateway compromise can expose every grant available to that process.
Decisions to make before launch
| Area | Decision the design must document |
|---|---|
| Provider credentials | Which workload owns each key, where it is stored, and how it is rotated |
| Tenant separation | How a request is prevented from selecting another tenant's provider account |
| Administration | Who may change routes, guardrails, plugins, or MCP connections |
| Egress | Which destinations the gateway may contact, including metadata and private services |
| Workload identity | What a compromised container could do in the cloud account |
| Recovery | How to revoke access and restart from a trusted configuration |
Separate the management interface from ordinary inference traffic. A user allowed to submit a prompt should not inherit the ability to upload code or redefine a route.
Start with a smaller permission set
Give the gateway a dedicated identity. Grant access to specific secrets and downstream resources. Remove provider credentials and cloud roles that were useful during development but are unnecessary for the deployed workload.
Make route configuration an approved artifact. If users need caller-selected destinations, put that feature behind explicit destination policy and egress controls.
Custom guardrails deserve the same scrutiny as application code. Ask where they compile, which process executes them, and whether every execution path uses the same confinement.
Evidence to bring to the review
Bring the effective role permissions, approved route list, tenant credential mapping, and a sample audit event for a configuration change. Then describe the expected blast radius if the gateway process is compromised.
If the answer includes cloud administration, every tenant's provider key, or unrestricted internal access, reduce that authority before adding another detection rule.