Scaling AI Gateways: How to Pool API Keys and Survive Rate Limits

When you first integrate Large Language Models (LLMs) into your product, everything works. You send a prompt, you get a response. But as your user base grows, you hit a hard wall: Provider Rate Limits.

Whether it's OpenAI's strict Tokens Per Minute (TPM) limits or Anthropic's Requests Per Minute (RPM) constraints, a single API key quickly becomes a bottleneck. The intuitive solution is to pool multiple API keys for the exact same model to artificially increase your total rate limit capacity.

However, managing a pool of provider API keys introduces an architectural shift. Let's explore why simple reverse proxies aren't enough, how to actually implement API key pooling, and the generic architecture required to do it safely.

The Catalyst: Hitting the Rate Limit Wall

Most major LLM providers enforce dual rate limits:

• RPM (Requests Per Minute): Caps the sheer volume of concurrent connections.
• TPM (Tokens Per Minute): Caps the actual compute throughput (often a mix of input and output tokens).

If you have a fast-growing AI application, hitting a 1M TPM limit is easy. A few heavy power users processing large documents can exhaust your quota in seconds, leaving the rest of your users staring at 429 Too Many Requests errors.

The logical step to overcome this is to provision multiple API keys (or multiple cloud deployments, such as several Azure OpenAI endpoints) and load balance traffic across them. If one key gives you 1M TPM, five keys give you 5M TPM.

The Naive Approach: Why Simple Load Balancers Fail

Your first instinct might be to stick Nginx, HAProxy, or a basic AWS ALB in front of your requests and configure a simple round-robin routing strategy across your pool of API keys.

This fails in production.

iWarning+

Standard reverse proxies track requests, not tokens.

Unlike traditional web traffic where requests have relatively uniform resource costs, LLM requests vary. Request A might ask a simple question consuming 50 tokens. Request B might pass in a massive PDF consuming 100,000 tokens.

If a naive round-robin balancer distributes these requests evenly by count, it will exhaust the token quotas of your underlying keys unevenly. One API key might hit its TPM limit instantly while another sits completely idle. To properly route traffic, the load balancer needs to understand the payload size, estimate token counts, and maintain state of how much capacity remains on each provider key.

The How-To: The 3-Layer AI Architecture

To actually build this, you need to abandon the idea of a single "smart proxy." Production LLM infrastructure splits responsibilities across three distinct layers.

1. The Control Plane (Source of Truth)

This layer owns the global state. It stores your Virtual Keys (user identities), user budgets, provider API keys, and live RPM/TPM counters. It is usually backed by Postgres (for durable billing/keys) and Redis (for fast token buckets).

2. The Routing Engine (Decision Maker)

This layer decides where the request should go. It checks the Control Plane to see which provider keys have remaining capacity, evaluates fallback logic, and selects the optimal upstream provider (e.g., choosing OpenAI Key #3 because it has the most TPM available).

3. The Gateway (Execution Layer)

This is the "dumb" data plane. It receives the payload, attaches the physical API key chosen by the router, forwards the request to the LLM, streams the Server-Sent Events (SSE) back to the client, and emits usage logs back to the Control Plane.

Implementing Distributed Rate Limits

How do you actually prevent overlapping requests from blowing past your limits when running multiple gateway nodes? You must externalize the quota check using pessimistic locking via Redis.

When a request arrives, the Control Plane intercepts it, estimates the token cost, and deducts that cost from a global Redis counter before forwarding the request to the Gateway. If the Redis counter hits zero, the request is blocked immediately.

Because Redis processes commands atomically, it doesn't matter if you have 1 or 100 gateway nodes processing requests simultaneously. Two concurrent requests cannot drain the same $1.00 budget. The Control Plane guarantees that the quota is never breached.

Next Steps: Choosing Your Implementation

You can build this 3-layer system entirely from scratch, but most teams adopt open-source AI gateways that implement parts (or all) of this pattern.

However, not all gateways handle global state the same way. In future posts, we'll dive into how this architectural requirement impacts your choices:

• LiteLLM: Bundles the Control Plane (via Redis) and the Router/Gateway together, making it highly capable of distributed, open-source rate limiting out-of-the-box.
• Bifrost: Acts as a fast Router/Gateway, but tracks state per-node. To scale it safely without paying for Enterprise clustering, you must build and attach your own external Control Plane.

For a deeper dive into the theory behind demand-side limits versus supply-side limits, explore our blueprint for architecting AI gateway budgets.