Rate Limiting with Python and Redis (GitHub)

 

import time

 
from client import get_redis_client
from exceptions import RateLimitExceeded
def rate_per_second(count):
    def _rate_per_second(function):
        def __rate_per_second(*args, **kwargs):
client = get_redis_client()
            key = f“rate-limit:{int(time.time())}“
            if int(client.incr(key)) > count:
                raise RateLimitExceeded
            if client.ttl(key) == –1:  # timeout is not set
                client.expire(key, 1)  # expire in 1 second
            return function(*args, *kwargs)
        return __rate_per_second
    return _rate_per_second
@rate_per_second(100)  # example: 100 requests per second
def my_function():
    pass  # do something
if __name__ == ‘__main__‘:
    success = fail = 0
    for i in range(2000):
        try:
            my_function()
            success += 1
        except RateLimitExceeded:
            fail += 1
        time.sleep(5/1000)  # sleep every 5 milliseconds
    print(f“Success count = {success}“)
    print(f“Fail count = {fail}“)

 

5.2.1 Storing counters in Redis

In order to update counters, we’ll need to store the actual counter information. For each counter and precision, like site hits and 5 seconds, we’ll keep a HASH that stores information about the number of site hits that have occurred in each 5-second time slice. The keys in the hash will be the start of the time slice, and the value will be the number of hits. Figure 5.1 shows a selection of data from a hit counter with 5-second time slices.

As we start to use counters, we need to record what counters have been written to so
that we can clear out old data. For this, we need an ordered sequence that lets us iterate
one by one over its entries, and that also doesn’t allow duplicates. We could use a LIST
combined with a SET, but that would take extra code and round trips to Redis. Instead,
we’ll use a ZSET, where the members are the combinations of precisions and names that
have been written to, and the scores are all 0. By setting all scores to 0 in a ZSET, Redis
will try to sort by score, and finding them all equal, will then sort by member name. This gives us a fixed order for a given set of members, which will make it easy to sequentially
scan them. An example ZSET of known counters can be seen in figure 5.2.

Figure 5.1A HASH that shows the number of web page hits over 5-second time slices around 7:40 a.m. on May 7, 2012
Figure 5.2A ZSET that shows some known counters

Git Hub: Akismet support for Python 2/3

Comment Check

akismet.check('192.168.1.3', 'Firefox / COMMENT USER AGENT', comment_author='King Arthur',
              comment_author_email='arthur@camelot.co.uk', comment_author_url='https://web.archive.org/web/20190910140301/http://www.camelot.co.uk:80/',
              comment_content='The Lady of the Lake, her arm clad in the purest shimmering samite, '
                               'held aloft Excalibur from the bosom of the water, signifying by divine'
                               ' providence that I, Arthur, was to carry Excalibur. That is why I '
                               'am your king.', referrer='http://camelot-search/?q=Peasant+Woman')

Submit Ham

akismet.submit_ham('192.168.1.12', 'FIREFOX / COMMENT USER AGENT', comment_author='Tim',
                   comment_author_email='tim@witch.co.uk',
                   comment_author_url='http://witch.co.uk',
                   comment_content="Look, that rabbit's got a vicious streak a mile wide!"
                                   "It's a killer!")

Submit Spam

akismet.submit_spam('192.168.1.4', 'FIREFOX / COMMENT USER AGENT', comment_author='Frenchman',
                    comment_author_email='frenchman@guy-de-lombard.fr',
                    comment_author_url='',
                    comment_content="You don't frighten us, English pig-dogs! Go and boil your "
                                    "bottoms, sons of a silly person! I blow my nose at you, "
                                    "so-called Ah-thoor Keeng, you and all your silly English "
                                    "K-n-n-n-n-n-n-n-niggits!")