Postgres Backup with Wal-e

The following are the steps I took to setup Wal-e 0.6.2 on Ubuntu 12.04.2 LTS and Postgres 9.1.9. After following the installation instructions, every minute Wal-e will make incremental backups to Amazon S3.

Installation

$ sudo apt-get install libevent-dev python-all-dev daemontools lzop pv postgresql-client
$ sudo pip install wal-e
$ umask u=rwx,g=rx,o=
$ mkdir -p /etc/wal-e.d/env
$ echo "secret-key-content" > /etc/wal-e.d/env/AWS_SECRET_ACCESS_KEY
$ echo "access-key" > /etc/wal-e.d/env/AWS_ACCESS_KEY_ID
$ echo 's3://some-bucket/directory/or/whatever' > /etc/wal-e.d/env/WALE_S3_PREFIX
$ sudo chown -R root:postgres /etc/wal-e.d

Added the following to the end of the file, /etc/postgresql/9.1/main/postgresql.conf:

wal_level = archive
archive_mode = on
archive_command = 'envdir /etc/wal-e.d/env /usr/local/bin/wal-e wal-push %p'
archive_timeout = 60

Restart postgres:

$ sudo service postgresql restart

PostgreSQL backup and restore service

pghoard is a PostgreSQL backup daemon and restore tooling for cloud object storages.

Features:

  • Automatic periodic basebackups
  • Automatic transaction log (WAL/xlog) backups (using either pg_receivexlog or archive_command)
  • Cloud object storage support (AWS S3, Google Cloud, OpenStack Swift, Azure, Ceph)
  • Backup restoration directly from object storage, compressed and encrypted
  • Point-in-time-recovery (PITR)
  • Initialize a new standby from object storage backups, automatically configured as a replicating hot-standby

Fault-resilience and monitoring:

  • Persists over temporary object storage connectivity issues by retrying transfers
  • Verifies WAL file headers before upload (backup) and after download (restore), so that e.g. files recycled by PostgreSQL are ignored
  • Automatic history cleanup (backups and related WAL files older than N days)
  • archive_sync tool for detecting holes in WAL backup streams and fixing them
  • Keeps statistics updated in a file on disk (for monitoring tools)
  • Creates alert files on disk on problems (for monitoring tools)