HireFire automatically “hires” and “fires” (aka “scales”) Delayed Job and Resque workers on Heroku. When there are no queue jobs, HireFire will fire (shut down) all workers. If there are queued jobs, then it’ll hire (spin up) workers. The amount of workers that get hired depends on the amount of queued jobs (the ratio can be configured by you).
This Is What You Build to Juggle 6,000 Tweets a Second
Twitter juggles tweets from more than 240 million people across the globe, with about 5,700 of these mini-messages sent every second, and this enormous stream of digital information gets stored on thousands of servers inside the company’s vast network of data centers. Because it contains so many types of data — from links and video to the bits of meta-data you don’t see — even a single tweet is spread across multiple machines. The trick lies in finding ways of grabbing all the right data from this sea of machines and rapidly delivering it to your phone or PC.
.. But about two years ago, Goffinet, Schuller, and Avital created Manhattan. According to the engineers, it lets the company expand across new machines much more easily than it could with Cassandra, and it lets them run both eventually consistent and strongly consistent applications.
Worker Dynos, Background Jobs and Queueing
Background jobs are key to building truly scalable web apps as they transfer both time and computationally intensive tasks from the web layer to a background process outside the user request/response lifecycle. This ensures that web requests can always return immediately and reduces compounding performance issues that occur when requests become backlogged.
A good rule of thumb is to avoid web requests which run longer than 500ms. If you find that your app has requests that take one, two, or more seconds to complete, then you should consider using a background job instead.
Postgres and Connection Pooling
A rule of thumb I’d use is if you have over 100 connections you want to look at something more robust
.. In contrast to the 5MB-ish standard connection to Postgres PG Bouncer has a 2kb per connection.