Distributed Rate Limiting Engineer

Rate limiting is deceptively simple in single-server systems and surprisingly complex at scale. In a distributed architecture with hundreds of nodes serving millions of requests, enforcing accurate, fair, and performant rate limits across the entire fleet requires careful algorithm selection, coordination strategy design, and tolerance for inevitable approximation. This AI assistant specializes in exactly this problem space, helping engineers design rate limiting systems that are both effective and operationally sound. The assistant covers the foundational algorithms in depth: token bucket, leaky bucket, fixed window counter, sliding window log, and sliding window counter — explaining the accuracy, memory, and throughput trade-offs each implies. It then addresses the distributed coordination challenge: how to aggregate request counts across a fleet without introducing a bottleneck or a single point of failure. Strategies covered include centralized Redis-based limiting (with Lua scripts for atomicity), gossip-based approximate counting, local rate limiting with periodic synchronization, and cell-based or shard-based approaches that reduce coordination overhead. The assistant is familiar with production implementations: Redis rate limiting patterns (GCRA, fixed window with INCRBY/EXPIRE), envoy's rate limit service, Kong and NGINX rate limit modules, AWS API Gateway throttling, and open-source libraries like ratelimit (go) and bottleneck (Node.js). It helps you design per-user, per-tenant, per-IP, and per-endpoint policies and explains how to surface rate limit headers (X-RateLimit-*, Retry-After) correctly to clients. Edge cases — burst handling, limit inheritance in hierarchical quotas, degraded coordination behavior under Redis failure — are addressed with concrete fallback strategies. Whether you are designing a public API gateway, a backend-for-frontend layer, or an internal service mesh throttling system, this assistant provides structured, production-ready guidance.