Index — Technomancy

Learning the basics of a language

written by rory, on Jul 1, 2009 10:06:49 PM.

Basic English is a simplified version of English that was created in the 1930s to make it easier for non-native speakers to learn the langauge. It has a list of about 1,500 words. I'm interested in learning more languages and one idea is to take this list of words, translate them into the desired language (eg Swahili) and then learn those words. That should give me a passing idea of that language. I've played around with a few rote learning programmes that help me learn a language. I'll have to blog about them.

PyLint plugin to catch if/elif blocks that don't have an 'else' clause

written by rory, on Mar 30, 2009 10:00:00 PM.

Recently I was tracking down a bug in some code. The source of the problem was an 'if' block that had a elif this, elif that, but it didn't have an 'else' clause. It should have had an else clause. It was going through a loop from a database and checking for certain conditions. It would set a variable based on that if block, since there was no else block, the variable was not getting updated in this iteration of the loop, causing a duplicate of the code. If there had been an 'else: assert False' it wouldn't have been a problem, because we could have caught it. After fixing the code here, I started to wonder about any other potential bugs like this were lurking elsewhere. It would be cool if I could analyse the code and see if there were any more if blocks without an else condition.

pylint is a programme that will scan your python source code for coding standards (variable names, unused imports, line length, etc). It's also extensible, so you can write your own plugin for it. this looked like the perfect tool to use. it turned out to be very straight forward to write a plugin that will scan for this kind of thing.

The code is stored in a git repository here. Anonymouse users can push to the 'mob' branch on this, so I welcome all submissions. It's quite easy to use, just call this command from the same directory as the missing_else.py file from the above repository. It'll generate warning messages for if blocks that have elif blocks but have no else blocks. pylint --load-plugins=missing_else sample.py

There are 2 options with this file. By default it will warn you if you have a if that has an elif and doesn't have an else clause. If you add the option "--warn_if_no_else=y" then it'll also warn you if you have a bare if clause that has no else clause. I don't enable this by default.

How to prefix each output line of a programme with the time and date

written by rory, on Mar 25, 2009 12:11:00 PM.

Many programmes produce logs, e.g. apache produces log files of webpages that people visit. One common feature of these log files is that each line has the time and date that the event happened at. This is very convienent because you can easily search for things that happened on a certain date or at a certain time.

Sometimes you might want to run a command that will print some output, and it might take a while. In cases like this it would be helpful to have a timestamp on each line. This is a simple little snippet you can put onto the end of a big command that will print timestamps.

The magic here is the 'read' built in. It will read the whole line (and store it in the shell variable line). However if there is no output yet, it will just sit there and the contents of the loop (which prints out the timestamp) will not be run. Since the time is only evaluated inside the loop, it will not update the time until it has a line. i.e. it will calculate a timestamp when a line is output by the first programme.

This is all you need to add to your command to get some nice output

$ command | while read line ; do NOW=$(date -Iseconds) ; echo -e "${NOW}\t${line}" ; done

Android Dev G1 on Meteor Ireland - Internet Settings

written by rory, on Mar 24, 2009 11:45:00 PM.

Meteor is the 3rd largest mobile phone provider in Ireland. They have a 3G service. I recently signed up with them so I could get their 3G internet service.

I bought an Android Dev G1 Phone. It's not locked to a certain network, so I can freely use any SIM card I want on it. However since I didn't get the phone from Meteor, I have to configure all the settings myself, including the APN to connect to the internet.

To set it up you need to enter these values. At the home screen, press menu, then 'Settings', then 'Wireless controls' then 'Mobile Networks' then 'Access Point Names'. At this screen press menu, then 'New APN'.

Fill in the following information:

The name can be anything, I have it set to "meteor"
APN: "data.mymeteor.ie"
Username: "my"
Password: "meteor"

then press Menu then Save.

After that you should have 3G/EDGE internet access on your android.

SQL - Updating each row with a grouped by total

written by rory, on Mar 23, 2009 10:45:00 PM.

Let's pretend you have an SQL table like this:

Name	Donation	Total Amount
Alice	10
Alice	12
Alice	8
Alice	9
Bob	2
Bob	3

and you want to fill in the column with the total for each person based on the Donation column You can use the following SQL command:

UPDATE table SET "Total Amount" = x.total from ( SELECT "Name", SUM("Amount") AS total FROM table GROUP BY Name ) AS x WHERE x.Name = Name;

A powerset generator in python

written by rory, on Mar 17, 2009 2:09:00 PM.

The powerset of a set is the set of all possible subsets of a set. i.e. for the list [1, 2, 3], the power set is [ [], [1], [2], [3], [1, 2], [1, 3], [2, 3], [1, 2, 3] ]. [wikipedia has more]. Generators in Python a powerful concept, and allow you to have lists that you can generate as you go along. So you don't need to calculate all the items in a list before using them. As you can see from the example above, the number of elements in a powerset of a list is much larger than the number of elements in the original list. If there are n items in the original list, then there are 2ⁿ items in the powerset. For example, the powerset of a 5 element list has 32 items, the powerset of a 10 element list has 1,024 items and so forth. Since powersets are so large, generators are very helpful here, since you don't have to generate (and store) a 1,024 element list before doing your calculation. Here is a simple generator function in python that will return (or 'yield') all the lists in the powerset of a list.

def powerset(seq):
    """
    Returns all the subsets of this set. This is a generator.
    """
    if len(seq) <= 1:
        yield seq
        yield []
    else:
        for item in powerset(seq[1:]):
            yield [seq[0]]+item
            yield item

Script to convert NMEA GPS traces to GPX

written by rory, on Mar 10, 2009 9:43:00 AM.

I have a Navi BGT-31 GPS reciever. I have it set to save things as NMEA, since that's a nice simple text format and it is easy to parse and save. It saves a different file every 2 hours. The following script will merge and convert NMEA TXT format files into a series of GPX tracks, with the date/time in the filename.

#! /bin/bash

FILES=$*

TMP_FILE=$(mktemp combined-traces.gpx.XXXXXX)
[ "$TMP_FILE" ] || exit 1

cat ${FILES} | gpsbabel -i nmea -f - -x sort,time -x discard,hdop=4,vdop=4 -x track,merge,sdistance=0.5k,title="track-%Y-%m-%d-%H:%M:%S" -x track,speed -o gpx -F ${TMP_FILE}

POINTS=$(xmlstarlet sel -N gpx=http://www.topografix.com/GPX/1/0  -t -m "/gpx:gpx" -v "count(//gpx:trkpt)" ${TMP_FILE})
NUM_TRACKS=$(xmlstarlet sel -N gpx=http://www.topografix.com/GPX/1/0  -t -m "/gpx:gpx" -v "count(./gpx:trk)" ${TMP_FILE})
echo "Found ${POINTS} trackpoints in ${NUM_TRACKS} tracks"

for TRACK in $(xmlstarlet sel -N gpx=http://www.topografix.com/GPX/1/0  -t -m "/gpx:gpx/gpx:trk" -v "./gpx:name" --nl ${TMP_FILE}) ; do
    echo "Extracting ${TRACK}.gpx..."
    gpsbabel -i gpx -f ${TMP_FILE} -x track,name=${TRACK} -o gpx -F ./${TRACK}.gpx
done

rm ${TMP_FILE}
echo "Done"

How to use pv and fifo to measure the speed of anything

written by rory, on Mar 4, 2009 5:34:00 PM.

A FIFO (first in first out) is a special kind of file on linux. It's basically a pipe that looks like a file. Linux Journal has a good introduction to named pipes pv is a very helpful little command line programme for monitoring the progress of data through a pipe, and showing how much has progressed, throughput rate and estimated time till it's done. If there's a long operation you can pipe the input through pv and see how far it's going and how long it's going to take. However some programmes don't/can't accept input from a pipe, you must use a file. I think mysqlimport is like this. You can use FIFOs and pv to get a progress bar. In one terminal, create the FIFO with

mkfifo pipe.fifo

Then pipe your big file into this FIFO using pv to get a status bar

pv bigfile > pipe.fifo

You won't see anything and you won't get your prompt back. FIFOs don't take from their input, unless something is trying to read that input. In another terminal, run your big command, using the fifo name as the filename. It will read from the 'file', which causes the pv to pipe stuff into it, which means the progress bar will appear.

bigcommand bigfile

When you're done, you can remove the FIFO 'file' since it's not really needed.

How to install SSH on an Android Dev G1

written by rory, on Jan 16, 2009 12:06:09 AM.

SSH is a way to remotely log onto other machines. It's vital to any sysadmins. ConnectBot is an excellent SSH client programme for Android Phones. It's very slick. The first time you start it up, there's a few screens that tell you the basic operations. It also tells you important things like how to do various Control-key combinations, which is not obvious. I think this is a great way to make your programme more usable. In some regions, you can download an old copy from the Android Market. However the USA has some restrictive laws about exporting cryptographic programmes. I'm in Ireland at the moment and so you can't get ConnectBot on the Android Market. You need to install it manually. From the ConnectBot website, download the latest version you can find. e.g. for me it was ConnectBot-svn-r174.apk. Save it somewhere. Plug your Android into your PC's USB. Open up a terminal and go to where you downloaded the Android SDK. Go into the tool directory, and execute the following the command:

./adb install /path/to/ConnectBot-svn-rXXXX.apk

So if I downloaded version 174, and saved it to my Desktop, I'd do the following command:

./adb install ~/Desktop/ConnectBot-svn-r174.apk

That's it! You should have ConnectBot available in the menu.

How to register an Android Dev G1 over WiFi

written by rory, on Jan 14, 2009 1:54:00 PM.

I got an Android Dev Phone. The first thing you have to do when you turn it on is register the phone to a Google Account. However I don't have a data plan with my phone, and could not register my new account, rendering my phone useless. The G1 has WiFi, and I was in a WiFi network, so I should have been able to use WiFi to connect to the google servers. By default you can't connect to WiFi. This is how you turn on WiFi. You'll need a computer running Ubuntu Linux. Install the Android SDK. Connect the phone to a PC via USB. According to this Google page, you have to create a file called /etc/udev/rules.d/50-android.rules and put this in it: So from a command line, run

$ sudo gedit /etc/udev/rules.d/50-android.rules

and put the following into it

SUBSYSTEM=="usb", SYSFS{idVendor}=="0bb4", MODE="0666"

Then make sure the permissions are a+rx:

sudo chmod a+rx /etc/udev/rules.d/50-android.rules

Unzip the SDK, and go to the tools directory in a terminal go to the 'tools' directory. The following command will show you the list of devices that are plugged in:

./adb devices
List of devices attached 
HT847GZ20615	device

If nothing shows up here, then you computer isn't detecting the phone. Make sure the phone is plugged in, and you have the correct file as above. Then restart the phone and go to the registration screen. Then you can connect to the phone's shell and manually launch the settings screen.

$ ./adb shell
$ su
# am start -a android.intent.action.MAIN -n com.android.settings/.Settings
#

This will bring up the Settings screen on your phone. From here you can tell it to connect to a certain WiFi network. After it's connected you can proceed and register the phone as normal.

Technomancy