Rate Limiting Strategies: Token Buckets vs Sliding Windows for AI APIs

[chawkian]

Rate Limiting Strategies for AI API Consumers

If you're hitting LLM APIs at any meaningful scale, rate limiting is going to be part of your life. Let's compare the two most common strategies and when to use each.

Token Bucket

• Allows short bursts up to the bucket capacity
• Refills at a steady rate
• Great for interactive apps where user requests come in spikes

Sliding Window

• Counts requests in a rolling time window
• Smoother throttling, no burst tolerance
• Better for background batch jobs with predictable throughput

Implementation Snippet (Node.js)

import { AIWave } from 'aiwave-sdk-node';

const client = new AIWave({ apiKey: process.env.AIWAVE_KEY });

// Simple sliding window limiter
class RateLimiter {
  constructor(maxRequests, windowMs) {
    this.max = maxRequests;
    this.window = windowMs;
    this.requests = [];
  }
  async acquire() {
    const now = Date.now();
    this.requests = this.requests.filter(t => now - t < this.window);
    if (this.requests.length >= this.max) {
      const wait = this.window - (now - this.requests[0]);
      await new Promise(r => setTimeout(r, wait));
    }
    this.requests.push(Date.now());
  }
}

const limiter = new RateLimiter(50, 60000); // 50 req/min

async function callLLM(prompt) {
  await limiter.acquire();
  return client.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: prompt }]
  });
}

Real-World Considerations

Most AI API providers (including AIWave) enforce both requests-per-minute and tokens-per-minute limits. Your client-side limiter should account for both dimensions — it's not just about request count.

Also worth thinking about: exponential backoff with jitter for 429 responses, and circuit breakers if you're fan-out across multiple models.

What's Your Approach?

Do you prefer client-side throttling or letting the server push back? Anyone using more advanced patterns like adaptive concurrency or priority queues? Share your setup! 🚀

---

Resources

• AIWave pricing
• AIWave developer blog
• AIWave Node.js SDK