One tenant's batch job could send thousands of requests a second. The API had no idea whose request it was serving, so everyone else waited behind them.
It worked for eleven months. Then a failover taught me what "eventually" means when the thing being counted is money.
One tenant's batch job could send thousands of requests a second. The API had no idea whose request it was serving, so everyone else waited behind them.
Three regions. A budget of 5 ms at p99 for the check itself. No new infrastructure beyond the Redis we already ran. The limit had to be per tenant and global, not per region.
A token bucket per tenant, checked in each region against a local Redis replica. Writes go to one primary and replicate out. Reads stay local, which is how the check stays fast.
A global lock would have made the count exact and added a cross-ocean round trip to every request. I chose to let each region trust a count that might be a few milliseconds old.
In a burst, regions can each admit requests before they hear about the others. We measured around 2% over the limit. For fairness that was fine. I wrote it down as accepted and moved on.
The whole check is one Lua script, so the read, the refill and the decrement happen atomically inside Redis. The Go side is a thin client with a timeout that fails open.
The primary failed. A replica that was behind got promoted, and it believed tenants had used far less than they had. For about forty seconds every region admitted traffic against counts that were already gone. One tenant got 141 requests on a limit of 100.
I had budgeted for small, steady inconsistency and never for a big, sudden one. Consistency is a budget. The real design question is who pays when you go over it. Now a promotion resets buckets to empty rather than trusting the replica.
Related note: understanding read skew →