jobtastic- Celery tasks plus celery-jQuery plugin

Jobtastic makes your user-responsive long-running Celery jobs totally awesomer. Celery is the ubiquitous python job queueing tool and jobtastic is a python library that adds useful features to your Celery tasks. Specifically, these are features you probably want if the results of your jobs are expensive or if your users need to wait while they compute their results.

Jobtastic gives you goodies like:

  • Easy progress estimation/reporting
  • Job status feedback
  • Helper methods for gracefully handling a dead task broker (delay_or_eager anddelay_or_fail)
  • Super-easy result caching
  • Thundering herd avoidance
  • Integration with a celery jQuery plugin for easy client-side progress display
  • Memory leak detection in a task run

Asynchronous Tasks With Django and Celery

When I was new to Django, one of the most frustrating things I experienced was the need to run a bit of code periodically. I wrote a nice function that performed an action that needed to run daily at 12am. Easy, right? Wrong. This turned out to be a huge problem to me since at the time I was used to “Cpanel-type” web hosting where there was a nice handy GUI for setting up cron jobs for this very purpose.

After much research, I found a nice solution – Celery, a powerful asynchronous job queue used for running tasks in the background. But this led to additional problems, since I couldn’t find an easy set of instructions to integrate Celery into a Django Project.

Django: Tracking Updates, Deletions, and Inserts

Logical Deletion

One of the features we wanted from the beginning was a default policy against deleting anything from the database. It helps with reversing mistakes and separates active data from stuff we want to keep but don’t want to see. The challenge in such a system is how to reliably keep all the objects that were soft-deleted out of our queries.

.. To track the model instance changes, we extended the queryset’s update(),delete(), create() methods as well as the model’s save() method. Anytime fields are changed through update() or save(), we generate a diff of what has changed and push it into the event database. Creation and deletion are also tracked. While it doesn’t track changes made directly in psql, it can alert us to data inconsistencies and point us to look at other logs.