Celery: Distributed Task Queue

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

jQuery and Long-Running Web App Processes: A Case Study

I was recently approached by a client’s system administrator with a small but interesting training/development project. The sys-admin, named Rod, had built a simple intranet web application that used a few PHP pages, running on an Amazon EC2 instance, to accept some user input and kick off multiple long-running server side QA deployment processes. He wanted to use Ajax to start the process as well as incrementally display its output, line-by-line, on the same web page. However, waiting for the entire process to finish to display its output was a poor user experience, and he wasn’t able to get an Ajax call to return any output incrementally over the lifetime of the request.

Web API performance: profiling Django REST framework

We’ve seen several cases of developers making these assumptions before they start building their API, and either discounting Django as not being fast enough for their needs, or deciding to not use a Web API framework such as Django REST framework because they ‘need something lightweight’.

I’m going to be making the case that Web API developers who need to build high performance APIs should be focusing on a few core fundamentals of the architecture, rather than concentrating on the raw performance of the Web framework or API framework they’re using.

.. the biggest performance gains for Web APIs can be made not by code tweaking, but by proper caching of database lookups, well designed HTTP caching, and running behind a shared server-side cache if possible.

.. 1. Get your ORM lookups right.

Given that database lookups are the slowest part of the view it’s tremendously important that you get your ORM queries right. Use .select_related() and.prefetch_related() on the .queryset attribute of generic views where necessary. If your model instances are large, you might also consider using defer() or only()to partially populate the model instances.

2. Your database lookups will be the bottleneck.

.. 3. Work on performance improvements selectively.

Remember that premature optimization is the root of all evil. Don’t start trying to improve the performance of your API until you’re in a position to start profiling the usage characteristics that your API clients exhibit. Then work on optimizing your views selectively, targeting the most critical endpoints first.