Technomancy

I am currently uploading some files to a remote computer, however my internet connection is not great. The connection intermittantly dies, killing the upload. I have to restart the upload as soon as I notice this. Luckily I’m using rsync, so files aren’t uploaded again.

I don’t want to have to restart the command. This little script, ‘do-till-done’ will repeatidly execute a command until it ends successfully (i.e. an exit status of 0). It used the libnotify notification system for GNOME to alert you when it needs to rerun a command. It waits 1 seconds between failures. Pressing Control-C will stop that iteration of the programme. To stop it fully, press Control-C while it’s in the sleep phase. i.e. press Control-C twice quickly.

Script:

#! /bin/bash

PROGNAME=$(basename $0)

COMMAND=$*

$COMMAND

while [ $? -ne 0 ] ;
do
    notify-send --urgency=low "${PROGNAME}" "${COMMAND} failed${WAIT_STR}"

    sleep 1

    $COMMAND
done

notify-send --urgency=normal "${PROGNAME}" "${COMMAND} finished"

This programme is in the public domain.

Sample usage:

$ do-till-done rsync -aPh *.jpg example.org:public_html/photos/

We all know that source code should be under a version control system. However databases are rarely version controlled (so to speak). Many times your application will depend on a certain structure of a database. If someone changes the structure your application can break.

This script will create mysql files for the table structure and tab separated value (TSV) for the values. TSV is used because it is the mysql default and it allows one to easily use many command line unix tools. The primary keys are needed so that the TSV files can be saved in order. Having them ordered means that if you only add one record, the diff if guaranteed to be only that line and not have all the other lines re-organised.

The database user and password is not saved because it’s intented that each developer would call this script after making a database change and would then commit the changes. This also means all your developers will see the change in the commit mails.

Change the DB_HOST, DB_BASE, TABLE_STRUCTURES, TABLE_VALUES and TABLE_PRIMARY_KEYS lines to suit your needs.

Assuming you had 3 tables, table1, table2 and table3 that you wanted to store the structure for. You want the values in table1 and table3 to be stored too. The primary key of table1 is ‘id’, and of table3 is ‘user_id’.

#! /bin/bash

## config

DB_HOST="hostname"
DB_BASE="database"

# we want to back up the structure of these tables
TABLE_STRUCTURES=( "table1" "table2" "table3" )

# we want to back up the values in these tables
TABLE_VALUES=( "table1" "table3" )

# The primary key in each table in TABLE_VALUES. n-th value here is the primary
# key of the n-th table in TABLE_VALUES
TABLE_PRIMARY_KEYS=( "id" "user_id" )

## end of config

# collect our values
echo -n "Username: "
read DB_USER

echo -n "Password: "

# read silently, ie don't echo the password
read -s DB_PASS

# force new line
echo ""

# read in the structure. TODO: spaces in tables might feck this up. TODO check [*] vs [@]
for TABLE_NAME in ${TABLE_STRUCTURES[*]} ; do
    FILE_NAME=${TABLE_NAME}.mysql

    if [ -e ${FILE_NAME} ] ; then
        OLD_MD5SUM=$(md5sum ${FILE_NAME})
    else
        OLD_MD5SUM=""
    fi;

    # TODO having pass on command line is bad...
    mysql -BNe "SHOW CREATE TABLE ${TABLE_NAME};" -u ${DB_USER} -h ${DB_HOST} -p${DB_PASS} ${DB_BASE} | sed -e "s/${TABLE_NAME}t//" -e 's/n/n/g' > ${FILE_NAME}

    # check if it's changed

    NEW_MD5SUM=$(md5sum ${FILE_NAME})

    if [ -z "${OLD_MD5SUM}" ] ; then
        echo "First time saving structure for ${TABLE_NAME}"
    else
        if [ "${OLD_MD5SUM}" != "${NEW_MD5SUM}" ] ; then
            # the structure has changed
            echo "The structure for ${TABLE_NAME} has changed"
        fi
    fi

done

# Now we get the values
for (( INDEX=0; $INDEX < ${#TABLE_VALUES[*]} ; INDEX=$(( $INDEX + 1 ))  )) ; do
    TABLE_NAME=${TABLE_VALUES[$INDEX]}
    TABLE_KEY=${TABLE_PRIMARY_KEYS[$INDEX]}

    # TODO: check TABLE_KEY exists, otherwise the SQL statement would be invalid

    FILE_NAME=${TABLE_NAME}.tsv
    if [ -e ${FILE_NAME} ] ; then
        OLD_MD5SUM=$(md5sum ${FILE_NAME})
    else
        OLD_MD5SUM=""
    fi;

    # TODO having pass on command line is bad...
    mysql -BNe "SELECT * FROM ${TABLE_NAME} ORDER BY ${TABLE_KEY};" -u ${DB_USER} -h ${DB_HOST} -p${DB_PASS} ${DB_BASE} > ${FILE_NAME}

    # check if it’s changed

    NEW_MD5SUM=$(md5sum ${FILE_NAME})

    if [ -z “${OLD_MD5SUM}” ] ; then
        echo “First time saving the values from ${TABLE_NAME}”
    else
        if [ “${OLD_MD5SUM}” != “${NEW_MD5SUM}” ] ; then
            # the structure has changed
            echo “The values in ${TABLE_NAME} have changed”
        fi
    fi

done

This script is copyrighted 2007 Rory McCann and released under the GNU General Public Licence v2 (or at your option a later version)

Shell scripts usually operate on a line by line basis, i.e. each line is a separate field. However MySQL uses “field1″,”field2″,.. type groups (e.g. for “SELECT * FROM table WHERE field IN ( “field1″,”field2″,.. );” ). This little alias will join all those lines together into something copy/pasteable into a MySQL query.

alias 'quotejoin=sed -e "s/^/"/" -e "s/$/",/" | tr -d "n" | sed "s/,$/n/"'

Example use:

$ cat test
line1
line2
line3
$ cat test | quotejoin
"line1","line2","line3"
$

I initially described how to automate an edubuntu install disc so you didn’t have to select the same options again and again.

Next I want to be able to install a PC with edubuntu by just plugging it into the network and booting off the network.

Basic Overview

When a computer does a network boot, a DHCP server gives it it’s network configuration, aswell as the path of the PXE (pre execution environment) that it is to boot. Luckily ubuntu comes with a PXE imagine prepared. It also needs to download the packages from an apt mirror. We can use the packages straight from the install CD for this. We can use one machine for all this. It’s only a matter of sticthing it all together.

Set up dhcpd

On Ubuntu install the dhcp3-server package You also need to tell the DHCP server (which will tell the clients) the path to the operating system to boot.

For example, here is section of my DHCPd configuration for the clients

subnet 192.168.2.0 netmask 255.255.255.0 {
    filename = "pxelinux.0";
    range 192.168.2.10 192.168.2.254;
    option subnet-mask 255.255.255.0;
    option broadcast-address 192.168.2.255;
    option routers 192.168.2.1;
    next-server 192.168.2.1;
}

All the install clients will get an IP address in the range 192.168.2.10 → 192.168.2.254. The ‘filename’ option tell the clients to download the file named ‘pxelinux.0′ over TFTP from this server and boot that OS. You’ll need the network interface to have the IP 192.168.2.1 for this.

Set up TFTP

TFTP (Trivial File Transfer Protocol) is used to send the base operating system image to the clients. In Ubuntu install the tftp-hpa package. By default the files in /var/lib/tftpboot will be server to the clients. So the clients will try to boot the image in /var/lib/tftpboot/pxelinux.0. Thanks to Conor Daly for assembling most of this. My /var/lib/tftptboot can be uncompressed into /var/lib/tftpboot to start you off.

Configure the PXE

When a machine boots the PXE, it uses /var/lib/tftpboot/pxelinux.cfg/default to give a little menu to the clients. This is the option for the installer:

label camarabuntu
	kernel camarabuntu/linux
    append initrd=camarabuntu/ubuntu-installer/i386/initrd.gz locale=en_IE console-setup/layoutcode=uk netcfg/get_hostname=camarabuntu netcfg/disable_dhcp=true netcfg/choose_interface=auto netcfg/get_ipaddress=192.168.2.10 netcfg/get_netmask=255.255.255.0 netcfg/get_gateway=192.168.2.1 netcfg/get_nameservers=192.168.2.1 netcfg/confirm_static=true preseed/url=http://192.168.2.1/camarabuntu.seed --

You’ll note there is lots of options for the installer in there. In earlier kernels there was a limit of about 8 options on the command line. In that case we would have had to make a custom initrd. Luckily in later kernels we can have up to 32 options, so we don’t need an initrd. We had to do something similar for the CD install. I’m not sure but I think the fact that the network is hardcoded in will break when there are more than one client. The important part is the preseed/url=http://192.168.2.1/camarabuntu.seed option which tells the installer to download that file and use that as the preseed file for the installer.

Preseed file

This is my camarabuntu preseed file. It’s very similar to the one for the CD. However we don’t need to do any network stuff, since that was set as a kernel boot option. Secondly we can’t do any network options here because if the installer is reading this file, it will have downloaded it and hence already have it’s network set up. Place this file in /var/www/camarabuntu.seed so it can be downloaded by the install clients.

# Set the installer language to Hiberno-English and UK keybaord layout
# Note: This are set before the cd is loaded, so they must be specified in the
# kernel options as so:
# locale=en_IE console-setup/layoutcode=uk
# These lines are incl
#d-i debian-installer/locale string ie_IE
#d-i console-setup/layoutcode string uk

# Install source
d-i     mirror/country		string enter information manually
d-i     mirror/http/hostname    string 192.168.2.1
d-i     mirror/http/directory   string /edubuntu
d-i     mirror/suite            string dapper
d-i     mirror/http/proxy       string

## Network
#d-i netcfg/choose_interface select auto
#d-i netcfg/disable_dhcp boolean true
#d-i netcfg/get_nameservers string 127.0.0.1
#d-i netcfg/get_ipaddress string 127.0.0.1
#d-i netcfg/get_netmask string 255.255.255.0
#d-i netcfg/get_gateway string 127.0.0.1
#d-i netcfg/confirm_static boolean true

# Adjust the default hostname.
d-i	netcfg/get_hostname	string camarabuntu

# Apt mirror
d-i	mirror/http/proxy	string	 

## Partitioning
# Put everything in one partition
d-i partman-auto/disk string /dev/hda

d-i partman-auto/choose_recipe 
       select All files in one partition (recommended for new users)

d-i partman/confirm_write_new_label boolean true
d-i partman/choose_partition 
       select Finish partitioning and write changes to disk
d-i partman/confirm boolean true

# Sync clock to UTC
d-i clock-setup/utc boolean true

# Set the timezone
d-i time/zone string Europe/Dublin

## user set up
# No root
d-i passwd/root-login boolean false
# But make a normal user
d-i passwd/make-user boolean true

# Set username & password
d-i passwd/user-fullname string Camara
d-i passwd/username string camara
d-i passwd/user-password password camara
d-i passwd/user-password-again password camara

# install grub in MBR, a handy default
d-i grub-installer/with_other_os boolean true

# Install the Edubuntu desktop and server.
tasksel	tasksel/first	multiselect edubuntu-desktop, edubuntu-server

# don't show us the 'Installing successful' dialog
d-i finish-install/reboot_in_progress note

# XServer set up.
xserver-xorg xserver-xorg/autodetect_monitor boolean true
xserver-xorg xserver-xorg/config/display/modes multiselect 1280x1024, 800x600, 640x480
xserver-xorg xserver-xorg/config/monitor/selection-method select medium
xserver-xorg xserver-xorg/config/monitor/mode-list select 1024x768 @ 60 Hz

Set up the ‘mirror’

In a CD install, the software packages are on the CD. When doing a network install you’re obviously going to be grabbing the software from the network cause there is no CD. Luckily you can use the same software packages from the install CD. In Ubuntu install the apache2 package, and start the apache webserver with the command

sudo /etc/init.d/apache2 start

Apache serves files from /var/www. I created a symlink /var/www/edubuntu to the cd-image folder (eg:

sudo ln -s /home/rory/camara/camarabuntu/cd-image /var/www/edubuntu

). Make sure the permissions are OK. All the files and folders are going to have to be world readable. (

chmod -R a+rX /home/rory/camara/camarabuntu

should do it).

Done

That’s it. ☺

References

• Conor Daly’s invaluable network boot set up

Files

• Contents of /var/lib/tftpboot

Changelog

• 21st August 2007 - Uploaded the correct tftpboot.tar file
• 20th August 2007 - Corrected link to the /var/lib/tftpboot

Well that was fun. My laptop battery died while in suspend. It managed to set the clock to 1904, which is before the Unix ‘epoch’ of 1st of January 1970. This caused all kinds of problems. When I tried to login in, Nautilus would not work and complained about connecting to the bonobo factory, it advised that I restart bonobo-activation-server. However it was not running. Starting it from the command line (i.e. /usr/lib/bonobo-activation/bonobo-activation-server) did nothing. When nautilus started up, bonobo-activation-server segfaulted. It looks like it can’t handle dates before the epoch.

However AFAIK, it’s not possible to set the date before the epoch with date(1). Only crazy stuff like killing the power while suspended will do it.

Based on my instructions on how to automate an edubuntu install CD, I have created the CD for Edubuntu 6.10 (aka Edgy Eft) and 5.10 (aka Breezy Badger). It’s available to download here:

Camarabuntu 6.10 - Edgy Eft
Camarabuntu 5.10 - Breezy Badger

I prefer the mysql command line for some mysql tasks. Other interfaces like PHPMyAdmin are annoying (readline is better than the stupid click to select textarea). To prevent myself making mistakes I have a read only user for my database (has global select and create temporay tables, that’s all). I’d like to know when I’m logged in as the read only user (and hence can do no damage), or something else.

You can use

SELECT CURRENT_USER();

or you can change the mysql prompt. The default is “mysql> “. Mine looks like this: “username@dbhost/database> “. Set your ~/.my.cnf to the following to get this behaviour.

[mysql]
prompt="u@h/d> "

References:

• MySQL documentation on mysql options

Trackpads on laptops are annoying if you’re using a mouse. While typing, you can accidentally tap it, causing the cursor to move.

Change your /etc/X11/xorg.conf file so the Synaptics Touchpad input device section looks like this:

Section "InputDevice"
	Identifier	"Synaptics Touchpad"
	Driver		"synaptics"
	Option		"SendCoreEvents"	"true"
	Option		"Device"		"/dev/psaux"
	Option		"Protocol"		"auto-dev"
	Option		"HorizScrollDelta"	"0"
    Option          "SHMConfig"
EndSection

The important line is the

    Option          "SHMConfig"

line.

After restarting X, you can disable the touchpad with this command:

synclient TouchpadOff=1

It can be enabled with

synclient TouchpadOff=0

Mac OSX disabled the touch pad when you plug a mouse in. GNOME should do this. There is an open bug about it.

The following python programme will detect when a mouse is plugged in (using HAL/Dbus) and disable the touchpad using the above command when a mouse is plugged in. If you plug in more than one mouse, crazy things might happen. You’ll need the python dbus package. Dump this somewhere, make it executable and add it to your start up programmes.

#! /usr/bin/python

import os

import dbus, gobject
from dbus.mainloop.glib import DBusGMainLoop

DBusGMainLoop(set_as_default=True)

system_bus = dbus.SystemBus()

def device_removed(device_name):
    global hal, initial_mice
    mice = hal.FindDeviceByCapability("input.mouse")
    # mice won't include the device that was removed, so we need to check
    # initial_mice
    if device_name in mice or device_name in initial_mice:
        enable_trackpad()

def device_added(device_name):
    global hal, initial_mice
    mice = hal.FindDeviceByCapability("input.mouse")
    if device_name in mice or device_name in initial_mice:
        disable_trackpad()

def enable_trackpad():
    os.system("synclient TouchpadOff=0")

def disable_trackpad():
    os.system("synclient TouchpadOff=1")

system_bus.add_signal_receiver(
    handler_function=device_removed,
    signal_name="DeviceRemoved",
    path="/org/freedesktop/Hal/Manager",
    dbus_interface="org.freedesktop.Hal.Manager" )

system_bus.add_signal_receiver(
    handler_function=device_added,
    signal_name="DeviceAdded",
    path="/org/freedesktop/Hal/Manager",
    dbus_interface="org.freedesktop.Hal.Manager" )

hal = system_bus.get_object('org.freedesktop.Hal', '/org/freedesktop/Hal/Manager')

initial_mice = hal.FindDeviceByCapability("input.mouse")

# see how many mice we have when we start up.
if len(initial_mice) == 1:
    # only one possible mouse so it's the trackpad.
    enable_trackpad()
else:
    disable_trackpad()

loop = gobject.MainLoop()
loop.run()

Sometimes when testing programmes that call other programmes, it’s important to be able to see what was called and what was on the stdin. This script prints all that info back to you.

#! /bin/bash
echo "script: args: $*"
while read line; do
    echo "script: stdin: $line"
done

exit 0

Example use:

$ cat > foo
Hello
this is line 2
$ cat foo | ./echor arg1 -n arg3
script: args: arg1 -n arg3
script: stdin: Hello
script: stdin: this is line 2

A lot of scripting languages like perl or python have good string and regular expression (regex) support. By using grep, you can have this in bash. Normally grep prints out the lines that match the regex. But with the -q option, it doesn’t print anything and exits as soon as it finds a match (or gets to the end). The return code will be 0 if it wasn’t found, or non-zero if it was found. This is ideal for including in an if clause. Remember to include ‘^’ and ‘$’ to ensure the regex is the only input.

eg:

if echo $VAR | grep -q '^d{3,10}$' ; then
  echo "VAR matches"
fi

Turns out you can do this in bash already but only in bash 3.0 or above.