Options to efficiently synchronize 1 million files with remote servers?
At a company I work for we have such a thing called “playlists” which are small files ~100-300 bytes each. There’s about a million of them. About 100,000 of them get changed every hour. These playlists need to be uploaded to 10 other remote servers on different continents every hour and it needs to happen quick in under 2 mins ideally
.. Since instant updates are also acceptable, you could use lsyncd.
It watches directories (inotify) and willrsyncchanges to slaves.
At startup it will do a fullrsync, so that will take some time, but after that only changes are transmitted.
Recursive watching of directories is possible, if a slave server is down the sync will be retried until it comes back.