Offsite Backups

Posted by admin, Wed Sep 05 14:01:00 UTC 2007

I have an external fireware enclosure that is now full of drives - 3.4TB in total. I use these drives for offsite backups. I connect to remote servers and rsync all the important stuff onto the drives on a 10 day cycle.

Thanks to some clever use of hard links and rsync I don’t need 10 times the disk space to do this though. It works like this -

Day 1 rsync remote directories to backup.1 on back drive

Day 2 Hard link backup.1 files to backup.2 rsync remote directories to backup.1

Day 3 Rename backup.2 to backup.3 Hard link backup.1 files to backup.2 rsync remote directories to backup.1

And so on. The number of backups in the cycle is configurable. The magic is in how rsync rebuilds a file that needs updating - it creates a new file and assembles it from the local file and the remote file and then removes the local file and replaces it with the new one - therefore breaking the hard link.

So if a file hasn’t changed it occupies the same space on the disk for each copy.

The script - below - takes 2 options - -b specifies the number of backups in a cycle and -s tells the script that you want a single update - this will simply refresh the newest backup - without doing any rolling back.

You must also supply a parameter to the script to tell it which backup you are running - this is used to mount the appropriate drive - achieved using drive labels.

Once the drive is mounted a file called backup.list is read - this file specifies the source server, the source path and the destination folder in the backup. i.e. 


webserver /var/www/ webserver/documentroot
dnsserver /etc/bind/ dnsserver/bind

The script

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124

#!/bin/ksh
#
# Script to mount usb drive, rsync various bits, umount
#

function log {
  D=$(date '+%Y/%m/%d %H:%M:%S')
  print -- "$D $*"
}

integer BACKUPS=5

TODO=0

OPTS="$*"

while getopts b:s ARG
do
  case $ARG in
    b) BACKUPS=$OPTARG
      ;;
    s) TODO=1
      ;;
    ?) print -u2 "Invalid argument"
      exit 1
      ;;
  esac
done

((OPTIND=OPTIND-1))
shift $OPTIND

if [[ $# != 1 ]]
then
  print -u2 "No host specified"
  exit 1
fi

HOST=$1

log "Backup Starting $OPTS"

MOUNTPOINT=/backups/backup.$HOST
mkdir $MOUNTPOINT > /dev/null 2>&1

# Find the device for the called HOST
if mount -L$HOST $MOUNTPOINT
then
  log "Disk for $HOST mounted on $MOUNTPOINT"
else
  log "Cannot find host backup volume $HOST"
  exit 1
fi

df -v $MOUNTPOINT

if [[ $TODO != 1 ]]
then
#roll back and recreate backup
  integer N
  integer M

  ((N=BACKUPS-1))
  while ((N>=2))
  do
    ((M=N+1))
    if [[ ! -d $MOUNTPOINT/backup.$N ]]
    then
      log "Creating sequential backup directory backup.$N"
      mkdir $MOUNTPOINT/backup.$N
    fi
    if [[ -d $MOUNTPOINT/backup.$M ]]
    then
      log "Removing backup directory backup.$M"
      rm -rf $MOUNTPOINT/backup.$M
    fi
    log "Moving backup.$N to backup.$M"
    mv $MOUNTPOINT/backup.$N $MOUNTPOINT/backup.$M
    RES=$?
    if [[ $RES != 0 ]]
    then
      STATUS="Failed"
    else
      STATUS="Complete"
    fi
    log "Sync $STATUS"
    ((N=N-1))
  done

  log "Creating sequential backup directory backup.2"
  mkdir $MOUNTPOINT/backup.2
  log "Duplicating latest backup into $MOUNTPOINT/backup.2"
  cp -al $MOUNTPOINT/backup.1/* $MOUNTPOINT/backup.2/.
else
  log "Refreshing latest backup only"
fi

while read SERVER SOURCE DESTINATION
do
  if [[ ! -d $MOUNTPOINT/backup.1/$DESTINATION ]]
  then
    if ! mkdir -p $MOUNTPOINT/backup.1/$DESTINATION
    then
      log -u2 "Failed to create $MOUNTPOINT/backup.1/$DESTINATION"
      continue
    fi
  fi

  log "Syncing $SERVER:$SOURCE to backup.1/$DESTINATION"
  rsync -az --delete --exclude="*\.mp3" $SERVER:$SOURCE $MOUNTPOINT/backup.1/$DESTINATION/.
  RES=$?
  if [[ $RES != 0 ]]
  then
      STATUS="Failed ($RES)"
    else
      STATUS="Complete"
  fi
  log "Sync $STATUS"
done < $MOUNTPOINT/backuplist
df -v $MOUNTPOINT

umount $MOUNTPOINT
rmdir $MOUNTPOINT
log "Backup for host $HOST complete"

Filed Under: Linux | Tags:

Comments

Leave a response

  1. orchard banks 06.05.08 / 19PM

    Nice Site! http://google.com

Have your say

A name is required. You may use HTML in your comments.




Recently

Categories