This post starts as a rant about webfaction, but somehow turns into a rave. I recently discovered (the hard way) that I can failover almost any site to a secondary host in a different data centre, all with a few scripts on a webfaction shared hosting account.
Piping gzip Command using rsync
You can pipe data over ssh instead of rsync; however your case will be more complicated.
Why Are Facebook, Digg, And Twitter So Hard To Scale?
Their caching tier services 120 million queries every second and it’s the core of the site. (2009)
.. But if you’ve ever wondered why Facebook has a 5,000 user limit on the number of friends, this is why. At a certain point it’s hard to make Pull on Demand scale.
Scalinig: Push on Change vs Pull on Demand
So we can use fast updates to speed up query execution. Moreover, use-cases typical for social applications are proven to be only scalable with push-on-change model (i.e. preliminary data propagation via updates with simple queries – the approach taken in this post) in comparison with pull-on-demand model (i.e. data are stored normalized and combined by queries on demand – classical relational approach). On push-on-change versus pull-on-demand read WHY ARE FACEBOOK, DIGG, AND TWITTER SO HARD TO SCALE?