Architecting AI Gateway Budgets: Virtual Keys, Provider Keys, and Distributed Quota Enforcement

As we established when discussing the challenges of pooling API keys, when building an AI Gateway to act as the primary interface between your users and upstream LLMs, routing requests is only half the battle. The far more difficult challenge is designing a product-level budgeting system.

If you don't design your quota system correctly, a sudden spike in concurrent requests can bypass your limits, bankrupting your upstream API accounts and causing sweeping 429 Too Many Requests outages for all your customers.

In this final post, we will walk through a blueprint for architecting a distributed AI quota engine.

The Architecture Split: Execution vs. Truth

The most common mistake when building an AI gateway is treating the proxy itself as the source of truth.

A production architecture must separate request execution from budget truth. The gateway can parse headers, manage streaming Server-Sent Events (SSE), and handle timeouts. But a dedicated budget module (or microservice) must own the authoritative view of user budgets, counters, and spend.

Demand-Side vs. Supply-Side Limits

A production AI gateway operates as a two-sided marketplace. A request is only valid if it passes checks on both sides of the equation.

1. Demand-Side Limits (The User)

This represents the constraints placed on the entity making the request.

• Identity: Who is this? (Mapped via a Virtual Key)
• Allowed Models: Can they use gpt-4o, or only gpt-3.5-turbo?
• Max Spend: Has this team exceeded their $500 monthly limit?
• Rate Limits: Are they exceeding their assigned 50 Requests Per Minute (RPM)?

2. Supply-Side Limits (The Provider)

This represents the constraints of your actual upstream infrastructure.

• Provider Keys: Which OpenAI/Anthropic keys are available?
• Capacity: Does sk-key-1 have enough Tokens Per Minute (TPM) remaining to fulfill this prompt?
• Cooldowns: Is this specific key currently locked out by a provider 429?

Virtual Keys Are the Budget Boundary

A Virtual Key is not merely an authentication token; it is the fundamental budget boundary.

In a multi-tenant system, the Virtual Key represents identity, model access, and spend attribution all at once. This highlights exactly why architectures that fail to synchronize Virtual Key state across multiple nodes (such as the open-source Bifrost deployments we analyzed previously) are flawed. If Virtual Key state is not globally coherent, your budget boundary simply does not exist under load.

The Request Flow of a Budget Module

To safely orchestrate these constraints, your budget module must execute a specific sequence of operations:

1. Resolve Virtual Key: Identify the user, team, or customer policy.
2. Check Model Access: Ensure the requested model is permitted for this Virtual Key.
3. Check Demand-Side Quota: Query Redis/Postgres to ensure the user has sufficient RPM/TPM and monetary budget.
4. Candidate Selection: Ask the routing layer for available Provider Keys.
5. Check Supply-Side Quota: Ensure the selected Provider Key has capacity.
6. Reserve Quota: Atomically reserve the estimated cost before forwarding the request.
7. Execute: Proxy the request to the LLM provider.
8. Settle: Calculate the actual tokens used from the provider's response headers, and update the final cost.
9. Persist: Write the durable spend ledger to Postgres.

Why Reservation Matters Under Concurrency

Step 6 (Reservation) is the linchpin of the entire system.

Imagine User X has $1.00 remaining in their budget. They launch a script that fires 100 concurrent requests at your gateway.

If your gateway uses a "read-only" budget check, all 100 requests will query the database at the exact same millisecond. All 100 requests will see "$1.00 remaining" and allow the request to proceed. A few seconds later, the LLM responds, the costs are settled, and User X has consumed $10.00 of compute, blowing past their hard limit by 10x.

iWarning+

To prevent this, the system needs atomic reservation. When a request arrives, the gateway must immediately deduct the estimated cost of the request (e.g., input_tokens + max_tokens) from the hot counter in Redis.

If the reservation pushes the budget below zero, the request is blocked. Once the request finishes, the gateway refunds any unused tokens back to the Redis counter. By executing the budget check and deduction atomically inside Redis, you completely eliminate the race condition.

Hard Limits, Soft Caps, and Settlement

Enforcing these limits requires distinct strategies depending on the metric:

• Hard RPM (Requests Per Minute): This is straightforward. You atomically increment a Redis counter before the route. If it exceeds the limit, block it.
• Hard-ish TPM (Tokens Per Minute): You must reserve estimated tokens before routing. Because LLM output tokens are unpredictable, you use the max_tokens parameter as the worst-case reservation.
• Soft Budgets (Total Spend): Vercel's AI Gateway documentation explicitly describes API key budgets as soft caps checked at the start of a request. You allow the "crossing request" (the one that pushes a user from $49.95 to $50.10) to complete, but immediately block all future requests during the settlement phase.

The Role of Redis and Postgres

A robust budgeting module splits responsibilities between ephemeral and durable datastores.

Redis owns the hot path:

• Live RPM/TPM token buckets.
• Atomic reservations for in-flight requests.
• Max parallel request counters.
• Short-lived provider cooldown locks.

Postgres owns the durable path:

• The authoritative list of Virtual Keys, Users, and Teams.
• Budget definitions and billing policies.
• The immutable spend ledger and reconciliation records.

Final Thesis

A production AI gateway is, at its core, a two-sided distributed quota engine. Virtual keys represent your demand-side identity and budget enforcement, while Provider keys represent your upstream capacity.

By using tools that separate durable and ephemeral state, like LiteLLM, to coordinate live reservations via Redis and record durable spend via Postgres, you can scale your AI infrastructure to handle heavy concurrency without bankrupting your API accounts. Any gateway architecture that cannot synchronize this state across its nodes, as we saw in our LiteLLM vs. Bifrost comparison, is unsafe for production.