Technomancy

Entries tagged “bash”

Show which directories have the most files (regardless of size)

written by rory, on Jan 21, 2010 11:51:00 AM.

I was rsync-ing a directory to remote host using an old version of rsync, so it had to count all the files on my local machine before it could start copying (newer versions of rsync start copying before reading all the files). However this was a large directory tree, with thousands and thousands of files, and it was taking rsync a long time to count. There are tools like ncdu to show how much disk space each folder is using. However in this case I want to reduce the number of files, regardless of their size, so ncdu is useless to me.

This bash snippet will show how many files & directories are in each folder, with the most 'populous' directories at the bottom.

find | cut -d/ -f2 | uniq -c | sort -n

How to prefix each output line of a programme with the time and date

written by rory, on Mar 25, 2009 12:11:00 PM.

Many programmes produce logs, e.g. apache produces log files of webpages that people visit. One common feature of these log files is that each line has the time and date that the event happened at. This is very convienent because you can easily search for things that happened on a certain date or at a certain time.

Sometimes you might want to run a command that will print some output, and it might take a while. In cases like this it would be helpful to have a timestamp on each line. This is a simple little snippet you can put onto the end of a big command that will print timestamps.

The magic here is the 'read' built in. It will read the whole line (and store it in the shell variable line). However if there is no output yet, it will just sit there and the contents of the loop (which prints out the timestamp) will not be run. Since the time is only evaluated inside the loop, it will not update the time until it has a line. i.e. it will calculate a timestamp when a line is output by the first programme.

This is all you need to add to your command to get some nice output

$ command | while read line ; do NOW=$(date -Iseconds) ; echo -e "${NOW}\t${line}" ; done 

Script to convert NMEA GPS traces to GPX

written by rory, on Mar 10, 2009 9:43:00 AM.

I have a Navi BGT-31 GPS reciever. I have it set to save things as NMEA, since that's a nice simple text format and it is easy to parse and save. It saves a different file every 2 hours. The following script will merge and convert NMEA TXT format files into a series of GPX tracks, with the date/time in the filename.
#! /bin/bash

FILES=$*

TMP_FILE=$(mktemp combined-traces.gpx.XXXXXX)
[ "$TMP_FILE" ] || exit 1

cat ${FILES} | gpsbabel -i nmea -f - -x sort,time -x discard,hdop=4,vdop=4 -x track,merge,sdistance=0.5k,title="track-%Y-%m-%d-%H:%M:%S" -x track,speed -o gpx -F ${TMP_FILE}

POINTS=$(xmlstarlet sel -N gpx=http://www.topografix.com/GPX/1/0  -t -m "/gpx:gpx" -v "count(//gpx:trkpt)" ${TMP_FILE})
NUM_TRACKS=$(xmlstarlet sel -N gpx=http://www.topografix.com/GPX/1/0  -t -m "/gpx:gpx" -v "count(./gpx:trk)" ${TMP_FILE})
echo "Found ${POINTS} trackpoints in ${NUM_TRACKS} tracks"

for TRACK in $(xmlstarlet sel -N gpx=http://www.topografix.com/GPX/1/0  -t -m "/gpx:gpx/gpx:trk" -v "./gpx:name" --nl ${TMP_FILE}) ; do
    echo "Extracting ${TRACK}.gpx..."
    gpsbabel -i gpx -f ${TMP_FILE} -x track,name=${TRACK} -o gpx -F ./${TRACK}.gpx
done

rm ${TMP_FILE}
echo "Done"