This is for my own benefit so I don’t forget :)

On remote git repository -

1
2
3

mkdir newapp.git
cd newapp.git
git --bare init

On Development platform -

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

rails newapp
cd newapp

git init

cat <<EOF > .gitignore
.DS_Store
log/*.log
tmp/**/*
db/*.sqlite3
coverage
doc/app/*
EOF

find . -type d -empty -exec touch {}/.gitignore 

git add .

git commit -a -m "Initial import." 

git remote add origin ssh://my.repository.com/git/newapp.git
git push origin master

note this is an old posting I’ve recreated for reference. The versions of files mentioned here will be out of date

I’m installing a dedicated rails server. This is the documentation on getting things working on Debian stable.

Firstly, I chose to install ruby and rubygems from source because the packages in Debian Stable are a little older than I would like.

First thing to do is get a development environment installed on Debian. The quickest/simplest way is to install the build-essential pseudo package.

aptitude install build-essential

Then we need zlib libraries for ruby

aptitude install zlib1g-dev

Download the ruby source code and extract it

tar zxvf ruby-1.8.6-p111.tar.gz

cd ruby-1.8.6-p111
./configure
make
sudo make install

Download ruby gems and extract it

tar zxvf rubygems-1.0.1.tgz
cd rubygems-1.0.1
ruby ./setup.rb

Install rails

gem install rails

Install msyql

aptitude install mysql-server

and client libraries

aptitude install libmysqlclient15-dev

gem install mysql

and that’s it done. | |

I’m installing a dedicated rails server. This is the documentation on getting things working on Debian stable.

Firstly, I chose to install ruby and rubygems from source because the packages in Debian Stable are a little older than I would like.

First thing to do is get a development environment installed on Debian. The quickest/simplest way is to install the build-essential pseudo package.

aptitude install build-essential

Then we need zlib libraries for ruby

aptitude install zlib1g-dev

Download the ruby source code and extract it

tar zxvf ruby-1.8.6-p111.tar.gz

cd ruby-1.8.6-p111
./configure
make
sudo make install

Download ruby gems and extract it

tar zxvf rubygems-1.0.1.tgz
cd rubygems-1.0.1
ruby ./setup.rb

Install rails

gem install rails

Install msyql

aptitude install mysql-server

and client libraries

aptitude install libmysqlclient15-dev

gem install mysql

and that’s it done.

I’m not a Gimp expert, and I am sure that you used to be able to fill to transparency in an obvious/easy way - however when I recently needed to do this I couldn’t figure it out.

In the end, I did it by doing the following -

• With the image open in Gimp, first add an alpha channel (Layer,Transparency,Add Alpha Channel)
• Add a layer mask - in the layers dialog, right click on the only layer present and choose Add Layer Mask. Initialise the mask to the layer’s alpha channel.
• Ensure the background mask is selected as the working layer by clicking on it.
• Select the bucket fill tool, choose black for the current foreground colour. Make sure that sample merged is ticked and flood fill the area you wish to erase.
• Once you have finished making all the transparent areas needed, right click on the layer again and choose Apply Layer Mask.

Job done!

I use the excellent gallery2 on my server and created a sub-album today for my Mum to upload her pictures to. I later thought it would be a good idea to have her gallery on her domain . The recommended way to have multiple galleries on one webserver involves per-site installations, and gallery2 makes this easy by only having one master install and then linking subsequent sites in. However I didn't want to do this as I like the idea of my Mums pictures being visible inside my gallery and also didn't want to worry about maintaining 2 different setups.

So, I needed a way to have web requests for the gallery on my Mum's site start off inside her gallery. I thought that maybe gallery2 would have an option for this - it seems a sensible idea. However, I guess because it already supports a fairly straightforward multi-site setup, it doesn't have such an option.

The solution to my problem was to use Apache's powerful rewriting engine to redirect all requests around the root of the gallery to the subalbum. However, gallery2 already makes extensive use of redirects so this isn't as straightforward as it might first appear.

The visible URLs that gallery2 uses are of the form /gallery/v/album and /gallery/d/picture, but these are rewritten to call main.php with parameters instead.

My first rewrite rule is pretty simple

RewriteRule ^/gallery/?$ /gallery/v/MumsGallery [R]

This handles the root entry point into the gallery. My Mum can give people a link to /gallery and it will rewrite into her subalbum.

However, there is a link inside the album to the parent album /gallery/main.php?g2_highlightId=900 which I also need to rewrite back into the subalbum.

This is done with the follow set of rules -

1
2
3

RewriteCond %{THE_REQUEST} /gallery/main.php
RewriteCond %{THE_REQUEST} !/gallery/main.php\?.*
Rewriterule ^.*$ /gallery/v/MumsGallery [R]

This basically says, if the URL is /main.php and it's not got any parameters then rewrite to the subalbum.

The [R] tells apache to restart from the top of the rewrite rule chain so the rewrites to gallery/v/MumsGallerys and be resolved to the main.php?g2_path=MumsGallery URL that gallery2 actually uses.

Often when I write scripts that do some sort of attentive processing I need to lock a particular resource. There are several way that you can do this but all revolve around filesystem lock files or directories.

The reason for this is because some file system calls are atomic - this means that it is guaranteed that the action can only occur successfully once.

You must be careful of falling into race conditions, for example code like this is flawed -

1
2
3
4
5
6
7
8
9
10

LOCKFILE=/tmp/lockfile

if [[ ! -r $LOCKFILE ]]
then
  echo $$ > $LOCKFILE
# Continue with process
else
# Lock file present
  exit
fi

There is a race condition here because the test is followed by the creating of the lock file - two separate processes could both detect that there is no lock file present and continue onwards.

A fail safe way of doing the about is below. Although I used to use this method I have moved on from this as I will explain further on.

1
2
3
4
5
6
7
8
9
10

LOCKFILE=/tmp/lockfile

if ( set -o noclobber; echo $$ > $LOCKFILE ) >/dev/null 2>&1
then
# Continue with process here
trap "rm $LOCKFILE" EXIT
else
# Lock file present
  exit
fi

This is simple and elegant and sufficient for most needs. The locking is done with the

( set -o noclobber; echo $$ > $LOCKFILE )

Note the trap to ensure that, whatever happens, the lock file will be removed when this process ends.

The method I use now is similar but instead of creating a file I create a directory. This is convenient for storing temporary files used by the process and a pid file. The mkdir call is again guaranteed to be atomic.

Below is an real life script that I use for spawning my offsite backup script. I have a list of separate backups that I need to run, and I want to run 3 processes simultaneously to make most use of the time and bandwidth available.

I’ll leave it up to you to follow the script and understand how it works.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

#!/bin/ksh
#

# Power up my external hard drives.
/usr/local/bin/heyu on enclosure

sleep 60

LOCKDIR=/tmp/backups

mkdir $LOCKDIR || exit
trap "rmdir $LOCKDIR" EXIT

cd /

function getalock {
        typeset LOCKQTY=$1

        integer N=1
        while ((N<=LOCKQTY))
        do
                if mkdir $LOCKDIR/$N >/dev/null 2>&1
                then
                        return $N
                fi
                ((N=N+1))
        done
        return 0
}

function removealock {
        N=$1
        rmdir $LOCKDIR/$N
}

function lockandgo {
        typeset CODE=$1
        shift
        typeset QTY=$1
        shift
        typeset MAILLIST=$*
        integer LOCKNUM

        while true
        do
                getalock $QTY
                LOCKNUM=$?
                if ((LOCKNUM==0))
                then
                                sleep 30
                                continue
                fi
                trap "removealock $LOCKNUM" EXIT
                echo $LOCKNUM /usr/local/bin/3gbackup  -b 10 $CODE $MAILLIST
                /usr/local/bin/3gbackup  -b 10 $CODE 2>&1 | mail -s "Backup $(date)" $MAILLIST
                return
        done
}

lockandgo SERVER1 3 andy@defsdoor.org &
lockandgo SERVER2 3 andy@defsdoor.org &
lockandgo SERVER3 3 andy@defsdoor.org &
lockandgo SERVER4 3 andy@defsdoor.org &
lockandgo SERVER5 3 andy@defsdoor.org &
lockandgo SERVER6 3 andy@defsdoor.org &
lockandgo SERVER7 3 andy@defsdoor.org &
lockandgo SERVER8 3 andy@defsdoor.org &

wait

sleep 300
/usr/local/bin/heyu off enclosure

I have about 12 external USB enclosures that I use for remote offsite backups. I connect to remote servers and rsync all the important stuff onto the drives on a 10 day cycle.

Thanks to some clever use of hard links and rsync I don’t need 10 times the disk space to do this though. It works like this -

Day 1 rsync remote directories to backup.1 on back drive

Day 2 Hard link backup.1 files to backup.2 rsync remote directories to backup.1

Day 3 Rename backup.2 to backup.3 Hard link backup.1 files to backup.2 rsync remote directories to backup.1

And so on. The number of backups in the cycle is configurable. The magic is in how rsync rebuilds a file that needs updating - it creates a new file and assembles it from the local file and the remote file and then removes the local file and replaces it with the new one - therefore breaking the hard link.

So if a file hasn’t changed it occupies the same space on the disk for each copy.

The script - below - takes 2 options - -b specifies the number of backups in a cycle and -s tells the script that you want a single update - this will simply refresh the newest backup - without doing any rolling back.

You must also supply a parameter to the script to tell it which backup you are running - this is used to mount the appropriate drive - achieved using drive labels.

Once the drive is mounted a file called backup.list is read - this file specifies the source server, the source path and the destination folder in the backup. i.e.

webserver /var/www/ webserver/documentroot
dnsserver /etc/bind/ dnsserver/bind 

The script

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124

#!/bin/ksh
#
# Script to mount usb drive, rsync various bits, umount
#

function log {
  D=$(date '+%Y/%m/%d %H:%M:%S')
  print -- "$D $*"
}

integer BACKUPS=5

TODO=0

OPTS="$*"

while getopts b:s ARG
do
  case $ARG in
    b) BACKUPS=$OPTARG
      ;;
    s) TODO=1
      ;;
    ?) print -u2 "Invalid argument"
      exit 1
      ;;
  esac
done

((OPTIND=OPTIND-1))
shift $OPTIND

if [[ $# != 1 ]]
then
  print -u2 "No host specified"
  exit 1
fi

HOST=$1

log "Backup Starting $OPTS"

MOUNTPOINT=/backups/backup.$HOST
mkdir $MOUNTPOINT > /dev/null 2>&1

# Find the device for the called HOST
if mount -L$HOST $MOUNTPOINT
then
  log "Disk for $HOST mounted on $MOUNTPOINT"
else
  log "Cannot find host backup volume $HOST"
  exit 1
fi

df -v $MOUNTPOINT

if [[ $TODO != 1 ]]
then
#roll back and recreate backup
  integer N
  integer M

  ((N=BACKUPS-1))
  while ((N>=2))
  do
    ((M=N+1))
    if [[ ! -d $MOUNTPOINT/backup.$N ]]
    then
      log "Creating sequential backup directory backup.$N"
      mkdir $MOUNTPOINT/backup.$N
    fi
    if [[ -d $MOUNTPOINT/backup.$M ]]
    then
      log "Removing backup directory backup.$M"
      rm -rf $MOUNTPOINT/backup.$M
    fi
    log "Moving backup.$N to backup.$M"
    mv $MOUNTPOINT/backup.$N $MOUNTPOINT/backup.$M
    RES=$?
    if [[ $RES != 0 ]]
    then
      STATUS="Failed"
    else
      STATUS="Complete"
    fi
    log "Sync $STATUS"
    ((N=N-1))
  done

  log "Creating sequential backup directory backup.2"
  mkdir $MOUNTPOINT/backup.2
  log "Duplicating latest backup into $MOUNTPOINT/backup.2"
  cp -al $MOUNTPOINT/backup.1/* $MOUNTPOINT/backup.2/.
else
  log "Refreshing latest backup only"
fi

while read SERVER SOURCE DESTINATION
do
  if [[ ! -d $MOUNTPOINT/backup.1/$DESTINATION ]]
  then
    if ! mkdir -p $MOUNTPOINT/backup.1/$DESTINATION
    then
      log -u2 "Failed to create $MOUNTPOINT/backup.1/$DESTINATION"
      continue
    fi
  fi

  log "Syncing $SERVER:$SOURCE to backup.1/$DESTINATION"
  rsync -az --delete --exclude="*.mp3" $SERVER:$SOURCE $MOUNTPOINT/backup.1/$DESTINATION/.
  RES=$?
  if [[ $RES != 0 ]]
  then
      STATUS="Failed ($RES)"
    else
      STATUS="Complete"
  fi
  log "Sync $STATUS"
done < $MOUNTPOINT/backuplist
df -v $MOUNTPOINT

umount $MOUNTPOINT
rmdir $MOUNTPOINT
log "Backup for host $HOST complete"