Mirroring cloud storage

From Arcmage Wiki
Jump to: navigation, search

What

The quest for an easy way to mirror the content from for example a local Dropbox-dir to a webserver/ftp is finally over: By using the GPL:d lftp we can pull it off. It is a very competent program that also supports FTPS, HTTP, HTTPS, HFTP, FISH, SFTP, FXP and even torrenting.


How To

  1. Use any cloud service you fancy. Set it so it auto-syncs one or more of your local dirs with the cloud. In our case we use Dropbox and all the devs share that account, so if one of us uploads or changes anything all of us will get the same thing changed on our local copies automagically.
  2. Setup an FTP server somewhere or have access to one that's already around. Make sure you have proper read & write access on the ftp user account you will be using. You will need the ability to read, write and delete both dirs and files.
  3. Tell lftp to mirror your local dirs (the ones your cloud service reads/write to) to the FTP. lftp is (surprisingly ;) an cmd line utility and can be custom tailored in many regards. Please consult the daunting MAN for some love.


Script

Here's what I used to mirror my local copy of the WTactics dropbox dev account to the WTactics public FTP from where people can fetch all our material:

#!/bin/bash
HOST='your.host.com'
USER='pingo'
PASS='weloveyou'

TARGETFOLDER='/dir/on/the/FTP'
SOURCEFOLDER='/home/snowdrop/.dropboxshit/Dropbox'

lftp -f "
open $HOST
user $USER $PASS
lcd $SOURCEFOLDER
mirror --continue --parallel=3 --log=/home/snowdrop/lftp.log --no-symlinks --use-cache --reverse --delete --exclude-glob=juice* --exclude-glob=*dropbox* --exclude-glob=.* --exclude-glob=.*/ --exclude-glob=w3storage/ --exclude-glob=uploads/ --exclude-glob=Thumbs.db --exclude-glob=*~ --exclude-glob=*.sla.autosave --verbose=3 $SOURCEFOLDER $TARGETFOLDER
bye
"

Ideally the script can be run using incron/inotify, but personally I prefer it as a normal cron since I don't want every small change to trigger an update of the FTP and rather see that it is done under more controlled forms.