Logging on the Freerunner

By default, the FreeRunner comes without a system log daemon, and rightfully so, because you don't want a daemon filling up /var, and a nightly cron job to rotate logs, on a mobile phone.

However, I do not want to give up logging. If there are bugs I want to investigate, I like to be able to look at the logs.

The solution: remote logging. Configure the phone to send all log info via UDP to my laptop, and configure my laptop to listen for UDP syslog messages from the phone only.

How to do it:

1. apt-get install rsyslog (see below how other logging daemons fail)
2. Add a /etc/hosts entry for the laptop on the phone: 192.168.0.200 base
3. Add a /etc/hosts entry for the phone on the laptop: 192.168.0.202 openmoko

4. Replace the phone's rsyslog rules to send everything to the laptop:

   ################
   ##### RULES ####
   ################
   
   
   # Everything goes to the main computer, via UDP, if available
   *.*     @base
   
   
   #
   # Emergencies are sent to everybody logged in.
   #
   *.emerg                         *
   
   

5. Tell the laptop's rsyslog to accept UDP messages from the phone only:

   # provides UDP syslog reception
   $ModLoad imudp
   $UDPServerRun 514
   $UDPServerAddress 192.168.0.200
   
   

That is all. When the phone is not connected, the UDP packets from the logger will go nowhere. When it gets connected to the laptop, a script is triggered that creates the 192.168.0.200 interface, rsyslogd will listen on that interface and the UDP packets will be logged.

This has the extra potential to be able to get error messages in case of bugs in the SD card driver, which I seem to hit from time to time.

Why rsyslog

sysklogd could receive the messages from the phone, but can only listen on all interfaces, and therefore requires me to set up firewall on my laptop in order to get packets from the phone only.

syslog-ng can and can be told to listen on a given interface, but it will refuse to start if that interface isn't available. Since the phone is on an hotplugged interface that comes and goes, this is totally unacceptable.

And finally, rsyslog is going to replace sysklogd both in Debian and in Fedora, so it's just as well a good excuse to switch.

Running apt on the FreeRunner

I've already mentioned that I'm running approx in the laptop and I configured the FreeRunner to access the laptop's cache. Here are the other customisations needed to have a decently working apt:

# cat /etc/apt/apt.conf.d/99freerunner 
APT::Install-Recommends "false";
Acquire::PDiffs "false";

The rationale is that recommends would bloat a system that is supposed to be small, and pdiff requires more CPU, memory and disk space/time than it actually saves in bandwidth.

Thanks to Michael Banck and Peter Palfrader for helping me to find out how to disable pdiffs.

How to read the Freerunner's accelerometers

This code has been take from moko_eightball by Jakob Westhoff: it just continuously prints the value of the three accelerometers.

#include <stdio.h>
#include <stdint.h>

void processInputEvents(FILE* in)
{
    int x = 0, y = 0, z = 0;
    while (1)
    {
        char padding[16];
        uint16_t type, code;
        int32_t value;

        // Skip the timestamp
        fread(padding, 1, 8, in);

        // Read the type
        fread(&type, 1, 2, in);

        // Read the code
        fread(&code, 1, 2, in);

        // Read the value
        fread(&value, 1, 4, in);

        switch( type )
        {
            case 0:
                switch( code )
                {
                    case 0:
                        fprintf(stdout, "x%d y%d z%d\n", x, y, z);
                        break;
                    default:
                        //warning( "Unknown code ( 0x%02x ) for type 0x%02x\n", code, type );
                        break;
                }
                break;
            case 2:
                switch ( code )
                {
                    case 0:
                        // Update to the new value
                        x = value;
                        break;
                    case 1:
                        // Update to the new value
                        y = value;
                        break;
                    case 2:
                        // Update to the new value
                        z = value;
                        break;
                    default:
                        //warning( "Unknown code ( 0x%02x ) for type 0x%02x\n", code, type );
                        break;
                }
                break;

            default:
                //warning( "Unknown type ( 0x%02x ) in accelerometer input stream\n", type );
                break;
        }


    }
}


int main()
{
    FILE* in = fopen("/dev/input/event2", "r");
    processInputEvents(in);
    fclose(in);
    return 0;
}

Polysms

Here is my first software designed for the FreeRunner: polysms. It's a commandline tool: you pass it a polygen grammar name and a phone number, and it will send a SMS to that phone number using the polygen output for that grammar as the SMS text:

# polyrun manager 0012345678

And here is the code, that works on the http://www.freesmartphone.org dbus framework:

#!/usr/bin/python

# (C) 2008 Enrico Zini
# Most bits of this are stripped from zhone, which is:
# (C) 2007 Johannes 'Josch' Schauer
# (C) 2008 Michael 'Mickey' Lauer <mlauer@vanille-media.de>
# (C) 2008 Jan 'Shoragan' Luebbe
# (C) 2008 Daniel 'Alphaone' Willmann
# (C) 2008 Openmoko, Inc.
# GPLv2 or later

from dbus import SystemBus, Interface
from dbus.exceptions import DBusException
import logging
logger = logging.getLogger( __name__ )

from dbus.mainloop.glib import DBusGMainLoop
DBusGMainLoop(set_as_default=True)

import gobject
import sys
from subprocess import Popen, PIPE


class Phone:
    def tryGetProxy( self, busname, objname ):
        try:
            return self.bus.get_object( busname, objname )
        except DBusException, e:
            logger.warning( "could not create proxy for %s:%s" % ( busname, objname ) )

    def __init__(self):
        try:
            self.bus = SystemBus()
        except DBusException, e:
            logger.error( "could not connect to dbus_object system bus: %s" % e )
            return False

        # Phone
        self.gsm_device_obj = self.tryGetProxy( 'org.freesmartphone.ogsmd', '/org/freesmartphone/GSM/Device' )

        if ( self.gsm_device_obj is not None ):
            self.gsm_device_iface = Interface(self.gsm_device_obj, 'org.freesmartphone.GSM.Device')
            self.gsm_sim_iface = Interface(self.gsm_device_obj, 'org.freesmartphone.GSM.SIM')
            self.gsm_network_iface = Interface(self.gsm_device_obj, 'org.freesmartphone.GSM.Network')
            self.gsm_call_iface = Interface(self.gsm_device_obj, 'org.freesmartphone.GSM.Call')
            self.gsm_test_iface = Interface(self.gsm_device_obj, 'org.freesmartphone.GSM.Test')

        # Main loop
        self.loop = gobject.MainLoop()

    def send(self, number, message):
        def onSent():
            print "SENT"
            self.loop.quit()
        def onStore(index):
            print "STORED AS", index
            self.gsm_sim_iface.SendStoredMessage(
                index,
                reply_handler=onSent,
                error_handler=self.onError
            )
        self.gsm_sim_iface.StoreMessage(
            number, message,
            reply_handler=onStore,
            error_handler=self.onError
        )

    def onError(self, result):
        print "ERROR", result

    def mainloop(self):
        self.loop.run()

if len(sys.argv) != 3:
    print >>sys.stderr, "Usage: %s grammarname phonenumber"
    sys.exit(1)

message = Popen(["/usr/bin/polyrun", sys.argv[1]], stdout=PIPE).communicate()[0]
number = sys.argv[2]

print "Sending to %s:" % number
print message

phone = Phone()
phone.send(number, message)
phone.mainloop()

Docking the FreeRunner into the laptop

Earlier in the day, I wrote:

So yes, my laptop can now be turned into a phone charger with networking, DNS and apt cache services. I shall look into hooking that script into dbus to have it run automatically when the phone is plugged and unplugged.

It's not dbus, it's udev, and I've managed to do it.

It's an udev rule:

# cat /etc/udev/rules.d/z60_openmoko_net.rules
ACTION=="add", SUBSYSTEM=="net", ATTRS{idVendor}=="1457", ATTRS{idProduct}=="5122", RUN+="/root/bin/share-openmoko"
ACTION=="remove", SUBSYSTEM=="net", INTERFACE="usb0", RUN+="/root/bin/share-openmoko"

And a script:

# cat bin/share-openmoko 
#!/bin/sh

if [ "$ACTION" == "add" ]
then
    ACTION=start
fi

if [ "$ACTION" == "remove" ]
then
    ACTION=stop
fi

INTERFACE=${INTERFACE:-"usb0"}
ACTION=${ACTION:-"$1"}

case "$ACTION" in
    start)
        logger -t openmoko "Connected, setting up network"
        iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE -s 192.168.0.200/29
        ifconfig usb0 192.168.0.200 netmask 255.255.255.248
        /etc/init.d/dnsmasq start
        echo 1 > /proc/sys/net/ipv4/ip_forward
        ;;
    stop)
        logger -t openmoko "Disconnected, bringing down network"
        echo 0 > /proc/sys/net/ipv4/ip_forward
        /etc/init.d/dnsmasq stop
        iptables -t nat -F POSTROUTING
        ifconfig usb0 down
        ;;
esac

The script has extra cruft that makes it double as a udev script and as a init.d style script, just because I didn't feel like abandoning the init.d style interface. However, udev handles it perfectly, so there's probably no use at all for the start/stop part.

This means that the interface and all supporting services will be brought up and down when the phone is connected/disconnected, and also when the phone is suspended/resumed. Of course, more fancy things can be plugged into the script, like syncing PIM info, file systems, turning on applets in panels, mounting phone file systems and whatnot.

Unpacking the new FreeRunner

I got myself a FreeRunner. Here are some notes from the first few days:

Available operating systems

The FreeRunner comes with the Om 2007.2 distribution: it works for basic phone things, and it has an opkg package manager that you can use to install all sort of extra software. Only issue: the SMS application was rather unstable with my SIM card.

I then tried Om 2008.8, which is a major new redesign, partly based on Qtopia. It's definitely more advanced on how it looks, with smooth animations all around, but I did not manage to get the GSM part to work, because I did not manage to get it to ask for a PIN.

I then tried plain Qtopia and I got what looks and feels like a properly working mobile phone. The only issue that it has is that I did not manage to make it suspend, so battery life is much shorter than it could be. Apparently, this should improve as kernel 2.6.26 reaches the phone. I kept Qtopia installed in the phone flash as a "stable" phone system.

And finally, Debian. Debian is based on the freesmartphone.org software stack, which is an attempt to create a UI agnostic DBUS frontend to the hardware that supports multiple applications running on top of it. It is a young project, but it can already drive a mobile phone in a useful way. It has a demo interface that is basically a showcase of what is implemented in the DBUS frontend, but does most of the basic things you need from a mobile phone, including taking and making phone calls. And then it has Debian behind, all of it. I need to buy a bigger microsd card.

Tips and tricks

Flashing things

1. apt-get install dfu-util
2. Start the Freerunner bios/bootloader, by holding down Power and then pressing AUX (standard bootloader), or by holding down AUX and then pressing Power (factory, unbrickable, read only fail safe bootloader).
3. dfu-util -l shows you a list of devices it can access. If you see more than once, you need to specify in all dfu-util commands which one you want, using -d USBID. In my case, I have to always use dfu-util -d 0x1d50:0x5119
4. To flash the kernel: dfu-util -a kernel -R -D /path/to/uImage
5. To flash the root file system: dfu-util -a rootfs -R -D rootfs_filename.jffs2
6. To flash the bootloader: dfu-util -a u-boot -R -D uboot_filename.bin
7. To flash the u-boot configuration: dfu-util -a u-boot_env -D env.new

You can also download all of these things from the FreeRunner by using -U instead of -D.

Networking via USB

All of the distributions I tried, by default configure the USB as a gadget with ethernet over usb. You can form a lan with it using the cdc_ether module, and you will find your phone preconfigured as 192.168.0.202 expecting to find a gateway at 192.168.0.200.

I made myself this script to start and stop networking with the phone:

#!/bin/sh

case "$1" in
    start)
        iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE -s 192.168.0.200/29
        ifconfig usb0 192.168.0.200 netmask 255.255.255.248
        /etc/init.d/dnsmasq start
        echo 1 > /proc/sys/net/ipv4/ip_forward
        ;;
    stop)
        echo 0 > /proc/sys/net/ipv4/ip_forward
        /etc/init.d/dnsmasq stop
        iptables -t nat -F POSTROUTING
        ifconfig usb0 down
        ;;
esac

Besides doing masquerading, it also brings up dnsmasq so that the phone can always find a DNS together with the router.

Another useful trick is to configure the phone to share the approx cache with the laptop:

deb http://192.168.0.200:9999/debian unstable main
deb http://192.168.0.200:9999/debian experimental main
deb http://pkg-fso.alioth.debian.org/debian unstable main

So yes, my laptop can now be turned into a phone charger with networking, DNS and apt cache services. I shall look into hooking that script into dbus to have it run automatically when the phone is plugged and unplugged.

Changing the ringtone in Debian

In case you don't like the default ringtone, this is how to change it:

1. vi /usr/share/python-support/fso-frameworkd/framework/subsystems/oeventd/receiver.py
2. look for def _play( self ):
3. change the codec in decoder = gst.element_factory_make, if needed. The list of available plugins is on the gstreamer website.
4. change the path in filesrc.set_property
5. /etc/init.d/fso-frameworkd restart and maybe /etc/init.d/zhone-session restart

Yes, the ringtone is currently hardcoded, but it does say it's a prototype after all. Or, if you prefer, it's fully configurable and the configuration can be found in /usr/share/python-support/fso-frameworkd.

Configuring the bootloader

You can connect to the u-boot bootloader via a serial terminal on /dev/ttyACM0. Type help and you will find that it can do a lot of things. The OpenMoko wiki has pages on the bootloader itself, its commands and the environment.

The environment is the configuration of the bootloader, similar somehow to /boot/grub/menu.lst. Unlike grub, you can edit the environment from within the bootloader and then save it using the saveenv command.

What I did:

1. A boot entry for Debian, with the kernel on an ext2 partition instead of fat: setenv menu_3 Boot from microSD (FAT+ext2): setenv bootargs \${bootargs_base} rootfstype=ext2 root=/dev/mmcblk0p2 rootdelay=5 \${mtdparts} ro\; mmcinit\; fatload mmc 1 0x32000000 \${sd_image_name}\; bootm 0x32000000
2. Stay in the bootloader until a choice has been made: setenv bootdelay -1
3. Don't power down the bootloader when idle: setenv boot_menu_timeout 99999
4. Always show the menu at boot: setenv stop_in_menu yes
5. saveenv

So now my phone dual boots, and I can choose if I want a more reliable phone now (Qtopia) or if I want to play with my future phone (Debian).

Configuring ssh for two host keys on the same host

Minor issue, but annoying: since both QTopia and Debian show up on 192.168.0.202, ssh will complain about changed host keys. Here is how to configure ssh to avoid the problem (in ~/.ssh/config):

Host debian
HostName 192.168.0.202
User root
HostKeyAlias debian

Host qtopia
HostName 192.168.0.202
User root
HostKeyAlias qtopia

Risotto ai funghi (e un po' di banana)

Avevo voglia di sperimentare, e in casa avevo delle banane. Cosa ci si può fare, con delle banane?

Entra in gioco http://www.foodpairing.be/

Questo sito raggruppa vari ingredienti in base alla comunanza delle sostanze chimiche che gli danno il sapore. E chi c'è vicino alla banana? I FUNGHI!

Facciamo quindi un risotto coi funghi: solito fondo di cipolla soffritta nel burro finché non diventa trasparente, e poi giú un pezzetto di banana tagliato a pezzettini sottili, a soffriggere anche lui e a caramellarsi un po'. Infine, qualche pezzetto di porcino secco rinvenuto in acqua tiepida.

Aggiungiamo poi il riso, lasciamolo soffriggere anche lui un po' nell'intingolo, e poi allunghiamo col brodo (io avevo un dado apposta per il risotto ai funghi comprato nel vicino negozietto di cose belle).

Niente sale, pepe, burro per mantecare, niente. Una volta cotto, l'ho solo lasciato a riposare per 5 minuti.

Il risultato è stato delizioso. "Ci hai messo la panna? Come fa a essere cosí cremoso?". Saporito ma non dolce. E la banana si sente che c'è, ma non si sente che è banana.

Da oggi mi sa che nel risotto ai funghi il mio ingrediente segreto sarà un pezzetto di banana.

Da notare che, come si legge in http://khymos.org/pairings.php, la banana sta anche bene col prezzemolo, e cosí i funghi. Mi son scordato il prezzemolo nel risotto in questo esperimento, ma ci sarà nel prossimo: ce l'abbiamo anche fresco in giardino. La stessa pagina parla anche di un probabile abbinamento molecolare cacao-funghi... chissà.

Pagine collegate:

• http://bressanini-lescienze.blogautore.espresso.repubblica.it/2008/07/02/accostamenti-sorprendenti/
• http://khymos.org/pairings.php
• TGRWT

Fagottini di pollo agli spinaci

È da tempo che cerco di capire come cucinare una buona bistecca, e finalmente ho trovato un sito di cucina che parla la mia lingua.

Giochiamo quindi con la Reazione di Maillard. Dopo un discreto successo con una bistecchina da quattro soldi, è venuto il momento di cimentarsi col pollo, che è l'unica carne che piace alla morosa.

La carne di pollo ha proteine, ma non abbastanza zuccheri perché avvenga la reazione di Maillard. Ergo, mariniamo la carne in qualcosa che contenga zuccheri.

Guggolando "pollo" e "marinata", esce questa bella ricetta "Petti di pollo ripieni al miele e aceto balsamico". La ricetta dice: "rosolatevi il pollo (3-4 minuti per lato, a fuoco medio)", ma io Maillard lo volevo guardare negli occhi e "fuoco medio" non mi bastava, e poi in Inghilterra non si trova lo speck, e io in casa avevo degli spinaci e non dell'"insalata Tatsoi", ergo, ho pistolato la ricetta come mio solito:

Arrosto di pollo ripieno di spinaci

Ingredienti:

• 3 petti di pollo a fette
• 3 fette di bacon magro, senza cotenna (siamo in Inghilterra...)
• 6-7 cubetti di spinaci surgelati
• 2 cucchiai di miele (meglio se d’Acacia)
• 2 cucchiai di aceto balsamico
• 1 cucchiaio di salsa di soia
• 1 cipollotto
• peperoncino secco sminuzzato
• olio, sale, pepe nero
• vari spicchi di aglio

Ho scongelato gli spinaci in un tegamino a fuoco basso, assieme a 4 o 5 spicchietti d'aglio schiacciati.

Nel frattempo ho fatto la marinata con miele, aceto balsamico, salsa di soia, il cipollotto tagliato finemente, uno spicchio o due d'aglio schiacciato, il peperoncino, due cucchiai d'olio, sale, pepe.

Ho poi spiattellato un po' i petti di pollo, ci ho messo sopra una fetta di pancetta, poi ho intonacato con uno strato di spinaci, arrotolato il tutto e legato con lo spago.

Pronti i fagotti li ho messi a mollo nella marinata. Li ho lasciati lí una buona mezz'oretta poi li ho girati e li ho lasciati lí un'altra mezz'oretta, in modo che si impregnassero e colorassero bene.

A questo punto, ho dato la molla al forno a 180° (per dopo) e ho messo sul fuoco una padella (io ho usato il wok antiaderente) con un pochino d'olio d'oliva.

Mi sono assicurato di non far danni con la fiamma vivace: la reazione di Maillard avviene oltre i 140°, il punto di fumo dell'olio d'oliva è dai 190° ai 240°, e quello del teflon dell'antiaderente è di 300°, quindi i margini ci sono.

Fiamma alta, olio caldo, giú il primo fagotto di pollo: due minuti per lato, col tegame coperto per limitare i danni degli schizzi. Ogni fagotto fatto da entrambi i lati l'ho poi messo in una pirofila, ci ho versato sopra un filo d'olio e ho messo tutto in forno per una 20ina di minuti, per stare nel sicuro perché, seppure a fiamma alta, 2 minuti per lato non mi sembravano abbastanza per cuocere il pollo e il bacon all'interno.

Tra un fagotto e l'altro vale la pena togliere dal tegame il grosso dei fondi e metterlo da parte, altrimenti a star lí per 3 fagotti su 2 lati c'è il rischio che bruci. Alla fine, col tegame bello incrostato, ci ho versato del vino, ho aggiunto i fondi messi da parte, e col fuoco basso e il cucchiaio di legno ho scrostato il tutto. Ho poi aggiunto un po' di zucchero per contrastare l'aspro del vino e ho lasciato restringere, dopodiché ho filtrato col colino e ho ottenuto una salsina deliziosa da cospargenere sui fagotti al momento di servire.

Purtroppo non ho la foto perché, vuoi l'aspetto vuoi il profumo, tutti e tre i fagotti sono spariti prima che ci venisse in mente di fare la foto.

Siccome era rimasto dell'unto invitante nel fondo della pirofila e il forno era ancora caldo, ci ho poi arrostito delle patate al forno. In Italia ci saranno 40 gradi, ma qui si fa fatica ad arrivare a 20.

Il tutto, annaffiato da una bottiglia di dolcetto del monferrato che trovammo tempo fa in sconto al supermercato: saporito com'era il pollo, un vino bianco non avrebbe avuto speranza.

Generating missing tangogps tiles

TangoGPS is nice, but it wants to download tiles. However, I already have a working mapnik setup, so why bother the OpenStreetMap servers with tile requests?

1. Set up a mapnik environment like I explained with gpsdrive, but use the -m switch to osb2pgsql.
2. svn co http://svn.openstreetmap.org/applications/rendering/mapnik/
3. Edit set-mapnik-env to configure it. Make sure that MAPNIK_TILE_DIR points to Maps/OSM as used by TangoGPS.
4. ./set-mapnik-env ./customize-mapnik-map > osm.xml
5. Edit generate_tiles.py to customise the area that you want to generate. Make sure that when setting bbox the first latitude is smaller than the second latitude, and the same for longitude.
6. Run ./set-mapnik-env ./generate_tiles.py and watch it paint your tiles. Note that if a tile has already been downloaded, it will not download it again.

Using OpenStreetMap maps with gpsdrive

Here's how I made it:

1. wget http://download.geofabrik.de/osm/europe/italy.osm.bz2
2. bunzip2 italy.osm.bz2 (this might not be needed)
3. setup mapnik; the instructions for sid worked on my lenny/testing system.
4. osm2pgsql -c /store/italy.osm (you should not use -m with gpsdrive)
5. start gpsdrive and turn on mapnik mode. Originally it did not work for me, until I deleted ~/.gpsdrive/osm.xml.

This makes gpsdrive not only more useful, but also completely Free.

Saving waypoints with Holux M-241

The Holux M-241 is a nice unit, but it looks like it cannot store waypoints while you're recording a track.

In fact, if you set it to display latitude and longitude, every time you press enter it stores a track point with the current location. However, after downloading the data with gpsbabel, they are indistinguishable from all the other track points.

If you are not taking a track, this is sufficient: your track will be made by all the waypoints you recorded.

Together with Riccio, another M-241 user, we noticed that if you are taking a track, with one trackpoint per second, then when you press enter in the lat/lon screen you get two track points in the same second: one from the logger, and one from your keypress.

The duplicate timestamp is just enough information to be able to distinguish a waypoint. Here is a little python script that will add a waypoint every time there are two trackpoints with the same timestamp.

#!/usr/bin/python

import xml.dom.minidom
from xml.dom.minidom import Node
import sys

def get_text(node):
    res = ""
    for n in node.childNodes:
        if n.nodeType == Node.TEXT_NODE:
            res += n.data
    return res.strip()

def get_val(node, name):
    for n in node.childNodes:
        if n.nodeName == name:
            return get_text(n)
    return None

doc = xml.dom.minidom.parse(sys.argv[1])

# Scan the document looking for duplicate timestamps
wpt = []
last_ts = None
for node in doc.getElementsByTagName("trkpt"):
    ts = get_val(node, "time")
    if last_ts != None and last_ts == ts:
        wpt.append(node)
    last_ts = ts

# Add the nodes with duplicate timestamps as waypoints
if len(wpt) > 0:
    for i in wpt:
        node = doc.createElement("wpt")
        for a in i.attributes.values():
            node.setAttribute(a.name, a.value)
        for n in i.childNodes:
            node.appendChild(n.cloneNode(True))
        doc.documentElement.appendChild(node)

doc.writexml(sys.stdout)

Playing with a Holux M-241

Here is how to download tracks from the Holux M-241:

gpsbabel -t -r -w -i m241 -f /dev/ttyUSB0 -o gpx -F `date +'%Y-%m-%d-%H%M%S'`.gpx

It might work also via bluetooth, but I have not tried yet.

Now, until M-241 support will be released in a stable version of gpsbabel, here is how to compile the version from CVS.

Get the sources:

cvs -d:pserver:anonymous@gpsbabel.cvs.sourceforge.net:/cvsroot/gpsbabel login
cvs -z3 -d:pserver:anonymous@gpsbabel.cvs.sourceforge.net:/cvsroot/gpsbabel co -P gpsbabel

Untar this to debianise the sources.

If it does to compile because of some errors in lmx.c, apply this patch.

Then you can install the resulting package and (hopefully) be happy.

Note, after downloading the logs, gpsbabel currently turns on logging. Here is a patch to disable that behaviour.

Finally, if you want to hack around a little on the unit, you can play with mtkbabel: the source code is simple, and most of the MTK protocol is implemented, so you can easily feed your own commands to the MTK. Documentation about the commands can be found here:

• http://spreadsheets.google.com/ccc?key=pyCLH-0TdNe-5N-5tBokuOA&t=2046668448833862841
• http://www.rc-cam.com/misc/PMTK_commands.pdf
• http://www.tronico.fi/OH6NT/docs/NMEA0183.pdf
• http://www.sparkfun.com/datasheets/GPS/MTK%20Packet_User%20Manual.pdf

The links are taken from a post in the GPSPasSion forum.

For a source of ideas of what commands you can send, you can look into the source code of BT747. For example, to set the M-241 to 2Hz fix, you can add this to mtkbabel:

packet_send('PMTK300,500,0,0,0.0,0.0');
$ret = packet_wait('PMTK001,300,3');

And if you want to set the MTK to an insane 5Hz fix rate, to take really fine grained gpx traces with your laptop, you can use this:

packet_send('PMTK300,200,0,0,0.0,0.0');
$ret = packet_wait('PMTK001,300,3');

Don't forget to set the serial speed to 38400 before talking with the unit:

stty 38400 < /dev/ttyUSB0

OpenStreetMap party at Kaohsiung, Taiwan

Apparently, yesterday we had the first OpenStreetMap event in Taiwan!

We met in a café/restaurant equipped with power plug, wireless network and overhead projector and we had a bit of an introduction, chat and lunch.

Then we split in groups and exploited the fact that the newly built underground (KMRT) system is still free of charge, to spread around and map around the stations.

Finally, we reconvened at someone's house to see how to put the data together, draw roads, tag and upload.

Highlights of the day:

• How to turn a serial GPS into a data logger with 6 hours battery life . Then attach it to your bike using magnets from broken hard drives. Totally rocks!
• Previous OpenStreetMap data was collected by only one person, who took the fancy new High Speed Rail from the opposite side of the country and joined the party. This also made discussion about standardising tags for Taiwan rather easy.
• A group of people appeared wielding a number of "totally insane in every regard" Garmin GPSMAP units: it turns out they are with a civil action group that goes around mapping historical trails, abandoned railroads, aboriginal routes and mountain crosses and so on. Apparently, they did not know about OpenStreetMap: hopefully they'll join in.

Technical bits:

• The eeePC was very popular, and very handy for going around storing tracks, as you can just chuck it in one bag. JOSM runs fine, although it could really use an interface redesign to fit in the small screen. In fact, it could really use an interface redesign to fit in the standard 1024x768 screen of my laptop.
• We could not use the tracks made with the Garmins because we did not know we had to do "Setup -> Map -> Lock On Road = Off" and it was on by default. Now we know it for next time.
• Something like a SirfStarIII really helps in a city made mainly of very tall buildings with lots of steel and glass. My Sony-based cheap gps receiver that worked ok in the Bolognese countryside was next to useless here, continously losing the fix and producing a crazy zigzagging track of doom, only useful to figure out big long straight roads.
• Geocorrelation of digital camera pictures rocks! Who needs to store waypoints when you can just take pictures with the digital camera and have them show up as waypoints in JOSM? The trick of taking a picture of the GPS time and use that to compute time offset is great. Also, we found it easier to just fire up gpscorrelate to do the geocorrelation rather than figuring out how the tools in JOSM work.

Issues to address:

• There is a strong need for a zh_TW translation plugin of JOSM; I'll try to find out how to do it and pass on the information to who can do it.
• Road names could be written either in English or in Chinese characters. Currently English has been used for the name tag because osmarender cannot render Chinese characters. There is some planining to create an OSM mirror in Taiwan which renders twice, and allows to choose the rendering language for the map. I will try to get a planet.osm extract for Taiwan that people can use to experiment with this; thanks to people in #osm for giving me names of people to contact. I will try later after Europe wakes up from this even-earlier-than-usual sunday morning.

Uploading gpsdrive tracks to openstreetmap

I've got some gpsdrive tracks and my area is blank on openstreetmap.

People pointed me at gpsbabel, and it took me a while to figure how it works. For the record, don't do any of this:

gpsbabel -i gpsdrive -o gpx -f track0000.sav -F track0000.gpx
gpsbabel -i gpsdrive -o gpx track0000.sav track0000.gpx

What you would have to do is this:

gpsbabel -i gpsdrive -f track0000.sav -o gpx -F track0000.gpx

However, it would choke on gpsdrive's "missing" points with all values set to 1001. You can grep them out, then gpsbabel would work, but openstreetmap would reject the data because the points have no timestamp: gpsbabel won't carry that on from the gpsdrive tracks to the GPX tracks.

The way to go is here, which contains a link to a tiny little perl script that will do the proper conversion for you:

./gpsdrive2gpx.pl track0000.sav > track0000.gpx

Those you can upload them to openstreetmap, at last.

OpenStreetMap party at Kaohsiung, Taiwan

Apparently, yesterday we had the first OpenStreetMap event in Taiwan!

We met in a café/restaurant equipped with power plug, wireless network and overhead projector and we had a bit of an introduction, chat and lunch.

Then we split in groups and exploited the fact that the newly built underground (KMRT) system is still free of charge, to spread around and map around the stations.

Finally, we reconvened at someone's house to see how to put the data together, draw roads, tag and upload.

Highlights of the day:

• How to turn a serial GPS into a data logger with 6 hours battery life . Then attach it to your bike using magnets from broken hard drives. Totally rocks!
• Previous OpenStreetMap data was collected by only one person, who took the fancy new High Speed Rail from the opposite side of the country and joined the party. This also made discussion about standardising tags for Taiwan rather easy.
• A group of people appeared wielding a number of "totally insane in every regard" Garmin GPSMAP units: it turns out they are with a civil action group that goes around mapping historical trails, abandoned railroads, aboriginal routes and mountain crosses and so on. Apparently, they did not know about OpenStreetMap: hopefully they'll join in.

Technical bits:

• The eeePC was very popular, and very handy for going around storing tracks, as you can just chuck it in one bag. JOSM runs fine, although it could really use an interface redesign to fit in the small screen. In fact, it could really use an interface redesign to fit in the standard 1024x768 screen of my laptop.
• We could not use the tracks made with the Garmins because we did not know we had to do "Setup -> Map -> Lock On Road = Off" and it was on by default. Now we know it for next time.
• Something like a SirfStarIII really helps in a city made mainly of very tall buildings with lots of steel and glass. My Sony-based cheap gps receiver that worked ok in the Bolognese countryside was next to useless here, continously losing the fix and producing a crazy zigzagging track of doom, only useful to figure out big long straight roads.
• Geocorrelation of digital camera pictures rocks! Who needs to store waypoints when you can just take pictures with the digital camera and have them show up as waypoints in JOSM? The trick of taking a picture of the GPS time and use that to compute time offset is great. Also, we found it easier to just fire up gpscorrelate to do the geocorrelation rather than figuring out how the tools in JOSM work.

Issues to address:

• There is a strong need for a zh_TW translation plugin of JOSM; I'll try to find out how to do it and pass on the information to who can do it.
• Road names could be written either in English or in Chinese characters. Currently English has been used for the name tag because osmarender cannot render Chinese characters. There is some planining to create an OSM mirror in Taiwan which renders twice, and allows to choose the rendering language for the map. I will try to get a planet.osm extract for Taiwan that people can use to experiment with this; thanks to people in #osm for giving me names of people to contact. I will try later after Europe wakes up from this even-earlier-than-usual sunday morning.

Glitches in the Matrix

Korean car with Taiwanese license plate (edited to anonymise it) over EU license plate with (Portuguese??) numbers on the right, and Korea as country code.

Italian pasta sold by a British supermarket, in Taiwan.

Also, "Messicani" is not a kind of Italian pasta. Google for it, and you'll only find it mentioned in British websites.

How to freak out a Frenchperson

The way to freak out an Italian, instead, is to show them a bottle of "Lambrini" in the UK.

Happy new year

A year ago we got in touch with various Taiwanese aboriginal tribes to try to start localisation efforts.

Thanks to the research the Taroko people did during 2007 and the prototype work of tonight, the Taroko people in Taiwan can see the computer calendar of the new year in their own language:

Meet the EeePC

Being in Taiwan, we swiftly got hold of an Eeepc.

Instead of installing Debian into it, we decided to keep the original system and see how it works. It's a Debian derivative, and the feeling inside a terminal window is quite familiar.

The boot is very fast. Two seconds after the video bios quickly shows on the screen, the X cursor appears. It's definitely worth having a look at how this devil boots.

The "Asus Launcher" is worth a look. IMHO it's nicer and more useful than the usual launcher menu that we get in Gnome or KDE 3, although it probably only makes sense on a small display. It replaces the desktop background, has tabs, no clutter and allows to launch applications. Turns out it's customisable as well.

What's on the system

KDE 3.4.2, with some applications renamed so that their names are more human. For example, konsole became console.

vim! \o/ But not emacs :)

mc! Someone out there wanted to make my life easier.

fbreader. I had never heard of it, but it's a very good discovery that I've now started to use it on my laptop as well.

Little howtos

To get to a terminal, hit Ctrl+T in the file manager, or Ctrl+Alt+T elsewhere.

The root password is the same as the user password.

To change the system language, I managed with a simple dpkg-reconfigure locales.

Ways they simplified the unix system

It's single user: I didn't find a way to create multiple users besides the terminal, and the login program does not ask for a username, only for the password.

The "win" key has a house painted on it, and it's used as "hide/show all applications" key. When all applications are minimised, the Asus launcher is visible instead of the X background: this behaviour basically turns the key into a sort of "run application" key. The key still works as a kind of shift, although it probably was not intended to.

The repository management is interesting. /etc/apt/sources.list contains:

deb http://update.eeepc.asus.com/p701 p701 main
deb http://update.eeepc.asus.com/p701/tw p701 main

which means they have a repository per eepc model and a subrepository per localised version.

The "Internet" group of applications has a Wikipedia toplevel application: it's nice to see the ecosystem of free software / free culture coming together to provide a nice user experience.

An extra link to the SD card mount point (besides the one in /mount) appears in the home directory automatically when the SD card is inserted. This means that when you do "save as" from all sorts of applications, the SD card is there, easy to reach. This helps if one decides not to use the internal flash for data, and just save everything in the SD card: I like doing this, as it allows me to quickly move the SD card with all the data between the EeePC and other computers.

Changes I made so far

Activate en_GB.UTF-8 via dpkg-reconfigure locales.

Add en_GB.UTF-8 to /etc/scim/global, to get SCIM input methods to work.

Little flaws

Virtual screens are enabled, so W+arrow switches virtual screen. The feeling you get if you hit W+arrow is that all your applications disappeared. This could be improved by having the vm keep the asus launcher at the bottom of the current virtual screen, instead of just at the bottom of the first screen. Or, to disable virtual screens by default.

It is possible to drag the lower panel around, maybe accidentally: that's another of our fancy default "features" that should be disabled by default.

It is also possible to remove applets from the applet bar by mistake: for example I wanted to disconenct the wireless, and I instead ended up quitting the wireless applet. Luckily, the next time I started the computer it magically came back.

~/.xsession-errors is continuously getting the useless stdout/stderr debugging flood of GUI apps. Noone bothers usually, except that in this case the file is on flash, where unneeded writes are also very much unwanted. I'm considering symlinking it to /dev/null, but ideally we should get GUI apps to only write out what is really important.

Battery charging doesn't show how long it is going to take until the battery is fully charged.

No capslock or numlock leds. This probably calls for disabling or remapping of capslock. Numlock is very hard to hit by accident, but capslock is.

Random thoughts

If you buy an eeepc, I really suggest you think of it a mass consumption appliance and stay on the original OS for a while. Most of what's in here is what we use everyday, just on a different context. Try to use it as an appliance and see if it is perfect, and if it isn't, try to find out what is missing. It is a fantastic way to find out important bits that are missing in Debian as well.

Also, if you're used to tailoring everything to yourself before starting to use a Linux system, this is a great way to try the usage experience that we can offer by default. The Firefox welcome page the first time you connect, for example, is surprisingly nice. Everything we know as doable comes a bit as as a surprise because this time someone has done it for us.

I wish that that someone can be invited to talk at the next Debconf: the possibility of having a look at the work that has been done in bending Debian to this nice little device is to me one of the most valuable things so far about the eeepc.

Help/About KDE/Credits

It's reachable by most applications, and says:

The development team would like to thank the following people and organizations for their contributions:

• the Debian Project,
• the GNU Project,
• the KDE Project,
• the Mozilla Project,
• the OpenOffice.org Project,
• the SAMBA Project,
• the X.Org Foundation,

Linus Torvalds and the other Linux kernel developers, and Free software developers around the world.

I'm using an appliance that is thanking me, and others like me: priceless!

Character list for the Amis language

We mapped the available glyphs and accents for the Amis language.

The letters in alphabetical order:

    a c d f ng h i k l m n o p r s t u w y

Everyone of them can get an acute or circumflex accent on top. ng can get a dot on top of the g.

The accents are literally on top: i would get the dot PLUS the accent on top.

Not all accented characters directly exist in Unicode; however Unicode developed various kinds of combination features to take care of these cases.

Then we need an input method that would insert ng instead of g and allow to type all the accent combinations.

Here is the full character set:

    a     á    â
    c     ć    ĉ
    d     d́    d̂
    f     f́    f̂
    ng    nǵ   nĝ  nġ
    h     h́    ĥ
    i     i̇́    i̇̂
    k     ḱ    k̂
    l     ĺ    l̂
    m     ḿ    m̂
    n     ń    n̂
    o     ó    ô
    p     ṕ    p̂
    r     ŕ    r̂
    s     ś    ŝ
    t     t́    t̂
    u     ú    û
    w     ẃ    ŵ
    y     ý    ŷ

Update: this character list has been improved and the good version is found in the Debian wiki.

The list is not displayed correctly with many fonts or rendering engines. Arne made a test page that explicitly sets a font that works.

The accents are not taken into account when sorting.

Uppercase letters are not used.

Note: the page has been updated to reflect further input from Unicode and Amis people.

Update: there is now a wiki page on the Debian wiki.

Character list for the Paiwan language

We mapped the available glyphs and accents for the Paiwan language.

The letters in alphabetical order:

a b c d e f h i j k l m n p q r s t u v w y z ḏ nġ ḻ ṟ ṯ

No uppercase.

Update: this character list has been improved and the good version is found in the Debian wiki.

All the characters are in Unicode except nġ, which already needs to be requested for the Amis script.

We need to design an input method to enter the underlined letters and the nġ.

Update: there is now a wiki page on the Debian wiki.

Amis and Paiwan input method and character set

Arne Götje (高盛華) created:

• a working SCIM input method for Amis
• a test text file with all the Amis characters
• a working SCIM input method for Paiwan
• a test text file with all the Paiwan characters
• a Browser test for Amis and Paiwan

The scripts, especially Amis, make heavy use of Unicode combination characters. They should display well at least with the Dejavu Sans font in many applications.

Try it out: if it displays correctly, you should see:

• accented letters instead of letters next to accents.
• i with both the dot and the accent.

Update: there is now a wiki page on the Debian wiki.

Creating a new locale

I'm currently in Cilamitay, in the east of Taiwan. There is a little meeting of Taiwanese Free Software people and people from the Amis, Taroko and Puyuma tribes, with the idea of starting localisation efforts for some aboriginal languages.

These are some of the issues we are going to discuss:

Language code

A new ISO standard (639-3) will hopefully be formalised in January that will include the language codes for the Taiwanese aboriginal tribes. We'll have to work some temporary solution, but there's good hope that it won't have to be temporary for long.

List of characters

Because of Christian missionary influence, both Amis and Taroko use a roman alphabet, with accents. We need to work out the complete list of character and accent combination, see if everything is in Unicode, see how they sort.

We then need to find a comfortable way to input them using the keyboards normally available here (English US layout): compose key? Dead keys? How about on Windows?

Womble2 on IRC tells me that on Windows one can works with MSKLC.

Technical terms and country list

We need to work out how to map terms that do not exist in the language.

Technical terms are usually borrowed from Japanese.

Names for all the countries in the world probably do not exist.

Translation interface

We need to find an easy to use interface to input the translations.

There is Rosetta.

There is Pootle. (Thanks to Christian Perrier for pointing me at it)

There is Webpot.

Update: there is now a wiki page on the Debian wiki.

Passing values to turbogears widgets at display time (the general case)

Last time I dug this up I was not clear enough in documenting my findings, so I had to find them again. Here is the second attempt.

In Turbogears, in order to pass parameters to arbitrary widgets in a compound widget, the syntax is:

form.display(PARAMNAME=dict(WIDGETNAME=VALUE))

And if you have more complex nested widgets and would like to know what goes on, this monkey patch is good for inspecting the params lookup functions:

import turbogears.widgets.forms
old_rpbp = turbogears.widgets.forms.retrieve_params_by_path
def inspect_rpbp(params, path):
    print "RPBP", repr(params), repr(path)
    res = old_rpbp(params, path)
    print "RPBP RES", res
    return res
turbogears.widgets.forms.retrieve_params_by_path = inspect_rpbp

The code for the lookup itself is, as the name suggests, in the retrieve_params_by_path function in the file widgets/forms.py in the Turbogears source code.

Linking to self in turbogears

I want to put in my master.kid some icons that allow to change the current language for the session.

First, all user-accessible methods need to handle a 'language' parameter:

@expose(template="myapp.templates.foobar")
def index(self, someparam, **kw):
    if 'language' in kw: turbogears.i18n.set_session_locale(kw['language'])

Then, we need a way to edit the current URL so that we can generate modified links to self that preserve the existing path_info and query parameters. In your main controller, add:

def linkself(**kw):
    params = {}
    params.update(cherrypy.request.params)
    params.update(kw)
    url = cherrypy.request.browser_url.split('?', 1)[0]
    return url + '?' + '&'.join(['='.join(x) for x in params.iteritems()])

def add_custom_stdvars(vars):
    return vars.update({"linkself": linkself})

turbogears.view.variable_providers.append(add_custom_stdvars)

(see the turbogears stdvars documentation and the cherrypy request documentation (cherrypy 2 documentation at the bottom of the page))

And finally, in master.kid:

<div id="footer">
  <div id="langselector">
    <span class="language">
      <a href="${tg.linkself(language='it_IT')}">
        <img src="${tg.url('/static/images/it.png')}"/>
      </a>
    </span>

    <span class="language">
      <a href="${tg.linkself(language='C')}">
        <img src="${tg.url('/static/images/en.png')}"/>
      </a>
    </span>
  </div><!-- langselector -->
</div><!-- footer -->

TurboGears RemoteForm tip

In case your RemoteForm misteriously behaves like a normal HTTP form, refreshing the page on submit, and the only hint that there's something wrong is this bit in the Iceweasel's error console:

Errore: uncaught exception: [Exception... "Component returned failure
code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsIXMLHttpRequest.open]"
nsresult: "0x80070057 (NS_ERROR_ILLEGAL_VALUE)"  location: "JS frame ::
javascript: eval(__firebugTemp__); :: anonymous :: line 1"  data: no]

the problem can just be a missing action= attribute to the form.

I found out after:

1. reading the TurboGears remoteform wiki: "For some reason, the RemoteForm is acting like a regular html form, serving up a new page instead of performing the replacements we're looking for. I'll update this page as soon as I figure out why this is happening."

2. finding this page on Google and meditating for a while while staring at it. I don't speak German, but often enough I manage to solve problems after meditating over Google results in all sorts of languages unknown or unreadable to me. I will call this practice Webomancy.

Python scoping

How do you create a list of similar functions in Python?

As a simple example, let's say we want to create an array of 10 elements like this:

a[0] = lambda x: x
a[1] = lambda x: x+1
a[2] = lambda x: x+2
...
a[9] = lambda x: x+9

Simple:

>>> a = []
>>> for i in range(0,10): a.append(lambda x: x+i)
...

...but wrong:

>>> a[0](1)
10

What happened here? In Python, that lambda x: x+i uses the value that i will have when the function is invoked.

This is the trick to get it right:

>>> a = []
>>> for i in range(0,10): a.append(lambda x, i=i: x + i)
...
>>> a[0](1)
1

What happens here is explained in the section "A Jedi Mind Trick" of the Instant Python article: i=i assigns as the default value of the parameter i the current value of i.

Strangely enough the same article has "A Note About Python 2.1 and Nested Scopes" which seems to imply that from Python 2.2 the scoping has changed to "work as it should". I don't understand: the examples above are run on Python 2.4.4.

Googling for keywords related to python closure scoping only yields various sorts of complicated PEPs and an even uglier list trick:

a lot of people might not know about the trick of using a list to box variables within a closure.

Now I know about the trick, but I wish I didn't need to know

Turbogears quirks when testing controllers that use SingleSelectField

Suppose you have a User that can be a member of a Company. In SQLObject you model it somehow like this:

    class Company(SQLObject):
        name = UnicodeCol(length=16, alternateID=True, alternateMethodName="by_name")
        display_name = UnicodeCol(length=255)

    class User(InheritableSQLObject):
        company = ForeignKey("Company", notNull=False, cascade='null')

Then you want to make a form that allows to choose what is the company of a user:

def companies():
    return [ [ -1, 'None' ] ] + [ [c.id, c.display_name] for c in Company.select() ]

class NewUserFields(WidgetsList):
    """Fields for editing general settings"""
    user_name = TextField(label="User name")
    companyID = SingleSelectField(label="Company", options=companies)

Ok. Now you want to run tests:

1. nosetests imports the controller to see if there's any initialisation code.
2. The NewUserFields class is created.
3. The SingleSelectField is created.
4. The SingleSelectField constructor tries to guess the validator and peeks at the first option.
5. This calls companies.
6. companies accesses the database.
7. The testing database has not yet been created because nosetests imported the module before giving the test code a chance to setup the test database.
8. Bang.

The solution is to add an explicit validator to disable this guessing code that is a source of so many troubles:

class NewUserFields(WidgetsList):
    """Fields for editing general settings"""
    user_name = TextField(label="User name")
    companyID = SingleSelectField(label="Company", options=companies, validator=v.Int(not_empty=True))

Turbogears form quirk

I had a great idea:

@validate(model_form)
@error_handler()
@expose(template='kid:myproject.templates.new')
def new(self, id, tg_errors=None, **kw):
    """Create new records in model"""
    if tg_errors:
        # Ask until there is still something missing
        return dict(record = defaults, form = model_form)
    else:
        # We have everything: save it
        i = Item(**kw)
        flash("Item was successfully created.")
        raise redirect("../show/%d" % i.id)

It was perfect: one simple method, simple error handling, nice helpful messages all around. Except, check boxes and select fields would not get the default values while all other fields would.

After two hours searching and cursing and tracing things into widget code, I found this bit in InputWidget.adjust_value:

# there are some input fields that when nothing is checked/selected
# instead of sending a nice name="" are totally missing from
# input_values, this little workaround let's us manage them nicely
# without interfering with other types of fields, we need this to
# keep track of their empty status otherwise if the form is going to be
# redisplayed for some errors they end up to use their defaults values
# instead of being empty since FE doesn't validate a failing Schema.
# posterity note: this is also why we need if_missing=None in
# validators.Schema, see ticket #696.

So, what is happening here is that since check boxes and option fields don't have a nice behaviour when unselected, turbogears has to work around it. So in order to detect the difference between "I selected 'None'" and "I didn't select anything", it reasons that if the input has been validated, then the user has made some selections, so it defaults to "The user selected 'None'". If the input has not been validated, then we're showing the form for the first time, then a missing value means "Use the default provided".

Since I was doing the validation all the time, this meant that Checkboxes and Select fields would never use the default values.

Hence, if you use those fields then you necessarily need two different controller methods, one to present the form and one to save it:

@expose(template='kid:myproject.templates.new')
def new(self, id, **kw):
    """Create new records in model"""
    return dict(record = defaults(), form = model_form)

@validate(model_form)
@error_handler(new)
@expose()
def savenew(self, id, **kw):
    """Create new records in model"""
    i = Item(**kw)
    flash("Item was successfully created.")
    raise redirect("../show/%d"%i.id)

If someone else stumbles on the same problem, I hope they'll find this post and they won't have to spend another two awful hours tracking it down again.

File downloads with TurboGears

In TurboGears, I had to implement a file download method, but the file required access controls so it was put in a directory not exported by Apache.

In #turbogears I've been pointed at: http://cherrypy.org/wiki/FileDownload and this is everything put together:

from cherrypy.lib.cptools import serveFile
# In cherrypy 3 it should be:
#from cherrypy.lib.static import serve_file

@expose()
def get(self, *args, **kw):
    """Access the file pointed by the given path"""
    pathname = check_auth_and_compute_pathname()
    return serveFile(pathname)

Then I needed to export some CSV:

@expose()
def getcsv(self, *args, **kw):
    """Get the data in CSV format"""
    rows = compute_data_rows()
    headers = compute_headers(rows)
    filename = compute_file_name()

    cherrypy.response.headers['Content-Type'] = "application/x-download"
    cherrypy.response.headers['Content-Disposition'] = 'attachment; filename="'+filename+'"'

    csvdata = StringIO.StringIO()
    writer = csv.writer(csvdata)
    writer.writerow(headers)
    writer.writerows(rows)

    return csvdata.getvalue()

In my case it's not an issue as I can only compute the headers after I computed all the data, but I still have to find out how to serve the CSV file while I'm generating it, instead of storing it all into a big string and returning the big string.

Passing values to turbogears widgets at display time

In turbogears, I often need to pass data to widgets at display time. Sometimes it works automatically, but sometimes, in cases like passing option lists to CheckBoxLists or number of repetitions in a RepeatingFieldSet, it doesn't.

All the examples use precomputed lists or pass simple code functions. In most of my cases, I want them computed by the controller every time.

Passing a function hasn't worked, as I did not find any obvious way to have the function know about the controller.

So I need to pass things the display() method of the widgets, but I could not work out how to pass the option list and default list for a CheckBoxList that is part of a WidgetsList in a TableForm.

On IRC came the answer, thanks to Xentac:

you should be able to...
    tableform.display(options=dict(checkboxname=[optionlist]))

And yes, it works. I can pass the default value as one of the normal form values:

    tableform.display(values=dict(checkboxname=[values]), options=dict(checkboxname=[optionlist]))

Turbogears i18n quirks

Collecting strings from .kid files

tg-admin i18n collect won't collect strings from your .kid files: you need the toolbox web interface for that.

Indentation problems in .kid files

The toolbox web interface chokes on intentation errors on your .kid files.

To see the name of the .kid file that causes the error, look at the tg-admin toolbox output in the terminal for lines like Working on app/Foo/templates/bar.kid.

What happens is that the .kid files are converted to python files, and if there are indentation glitches they end up in the python files, and python will complain.

Once you see from the tg-admin toolbox standard error what is the .kid file with the problem, edit it and try to make sure that all closing tags are at the exact indentation level as their coresponding opening tags. Even a single space would matter.

Bad i18n bug in TurboKid versions earlier than 1.0.1

faide on #turbogears also says:

It is of outmost importance that you use TurboKid 1.0.1 because it is the first version that corrects a BIG bug regarding i18n filters ...

The version below had a bug where the filters kept being added at each page load in such a way that after a few hundreds of pages you could have page loading times as long as 5 minutes!

If one has a previous version of TurboKid, one (and only one) of these is needed:

• Patch Kid with the patch at http://www.kid-templating.org/trac/ticket/203
• Patch TurboKid with the patch at http://trac.turbogears.org/ticket/1301
• Upgrade to TurboGears 1.0.2.2, which also works like a charm with python 2.5, which is a performance boost.

So, in short, all i18n users should upgrade to TurboGears 1.0.2.2 or patch TurboKid using http://trac.turbogears.org/ticket/1301.

Quirks when overriding SQLObject setters

Let's suppose you have a User that is, optionally, a member of a Company. In SQLObject you model it somehow like this:

    class Company(SQLObject):
        name = UnicodeCol(length=16, alternateID=True, alternateMethodName="by_name")
        display_name = UnicodeCol(length=255)

    class User(InheritableSQLObject):
        company = ForeignKey("Company", notNull=False, cascade='null')

Then you want to implement a user settings interface that uses a Select box to choose the company of the user.

For the Select widget to properly handle the validator for your data, you need to put a number in the first option. As my first option, I want to have the "None" entry, so I decided to use -1 to mean "None".

Now, to make it all blend nicely, I overrode the company setter to accept -1 and silently convert it to a None:

    class User(InheritableSQLObject):
        company = ForeignKey("Company", notNull=False, cascade='null')

        def _set_company(self, id):
            "Set the company id, using None if -1 is given"
            if id == -1: id = None
            self._SO_set_company(id)

In the controller, after parsing and validating all the various keyword arguments, I do something like this:

            user.set(**kw)

Now, the overridden method didn't get called.

After some investigation, and with the help of NandoFlorestan on IRC, we figured out the following things:

1. That method needs to be rewritten as _set_companyID:

           def _set_companyID(self, id):
               "Set the company id, using None if -1 is given"
               if id == -1: id = None
               self._SO_set_companyID(id)
   
   

2. Methods overridden in that way are alsop called by user.set(**kw), but not by the User(**kw) constructor, so using, for example, a similar override to transparently encrypt passwords would give you plaintext passwords for new users and encrypted passwords after they changed it.

Passing values to turbogears widgets at display time (the general case)

Last time I dug this up I was not clear enough in documenting my findings, so I had to find them again. Here is the second attempt.

In Turbogears, in order to pass parameters to arbitrary widgets in a compound widget, the syntax is:

form.display(PARAMNAME=dict(WIDGETNAME=VALUE))

And if you have more complex nested widgets and would like to know what goes on, this monkey patch is good for inspecting the params lookup functions:

import turbogears.widgets.forms
old_rpbp = turbogears.widgets.forms.retrieve_params_by_path
def inspect_rpbp(params, path):
    print "RPBP", repr(params), repr(path)
    res = old_rpbp(params, path)
    print "RPBP RES", res
    return res
turbogears.widgets.forms.retrieve_params_by_path = inspect_rpbp

The code for the lookup itself is, as the name suggests, in the retrieve_params_by_path function in the file widgets/forms.py in the Turbogears source code.

Linking to self in turbogears

I want to put in my master.kid some icons that allow to change the current language for the session.

First, all user-accessible methods need to handle a 'language' parameter:

@expose(template="myapp.templates.foobar")
def index(self, someparam, **kw):
    if 'language' in kw: turbogears.i18n.set_session_locale(kw['language'])

Then, we need a way to edit the current URL so that we can generate modified links to self that preserve the existing path_info and query parameters. In your main controller, add:

def linkself(**kw):
    params = {}
    params.update(cherrypy.request.params)
    params.update(kw)
    url = cherrypy.request.browser_url.split('?', 1)[0]
    return url + '?' + '&'.join(['='.join(x) for x in params.iteritems()])

def add_custom_stdvars(vars):
    return vars.update({"linkself": linkself})

turbogears.view.variable_providers.append(add_custom_stdvars)

(see the turbogears stdvars documentation and the cherrypy request documentation (cherrypy 2 documentation at the bottom of the page))

And finally, in master.kid:

<div id="footer">
  <div id="langselector">
    <span class="language">
      <a href="${tg.linkself(language='it_IT')}">
        <img src="${tg.url('/static/images/it.png')}"/>
      </a>
    </span>

    <span class="language">
      <a href="${tg.linkself(language='C')}">
        <img src="${tg.url('/static/images/en.png')}"/>
      </a>
    </span>
  </div><!-- langselector -->
</div><!-- footer -->

TurboGears RemoteForm tip

In case your RemoteForm misteriously behaves like a normal HTTP form, refreshing the page on submit, and the only hint that there's something wrong is this bit in the Iceweasel's error console:

Errore: uncaught exception: [Exception... "Component returned failure
code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsIXMLHttpRequest.open]"
nsresult: "0x80070057 (NS_ERROR_ILLEGAL_VALUE)"  location: "JS frame ::
javascript: eval(__firebugTemp__); :: anonymous :: line 1"  data: no]

the problem can just be a missing action= attribute to the form.

I found out after:

1. reading the TurboGears remoteform wiki: "For some reason, the RemoteForm is acting like a regular html form, serving up a new page instead of performing the replacements we're looking for. I'll update this page as soon as I figure out why this is happening."

2. finding this page on Google and meditating for a while while staring at it. I don't speak German, but often enough I manage to solve problems after meditating over Google results in all sorts of languages unknown or unreadable to me. I will call this practice Webomancy.

Python scoping

How do you create a list of similar functions in Python?

As a simple example, let's say we want to create an array of 10 elements like this:

a[0] = lambda x: x
a[1] = lambda x: x+1
a[2] = lambda x: x+2
...
a[9] = lambda x: x+9

Simple:

>>> a = []
>>> for i in range(0,10): a.append(lambda x: x+i)
...

...but wrong:

>>> a[0](1)
10

What happened here? In Python, that lambda x: x+i uses the value that i will have when the function is invoked.

This is the trick to get it right:

>>> a = []
>>> for i in range(0,10): a.append(lambda x, i=i: x + i)
...
>>> a[0](1)
1

What happens here is explained in the section "A Jedi Mind Trick" of the Instant Python article: i=i assigns as the default value of the parameter i the current value of i.

Strangely enough the same article has "A Note About Python 2.1 and Nested Scopes" which seems to imply that from Python 2.2 the scoping has changed to "work as it should". I don't understand: the examples above are run on Python 2.4.4.

Googling for keywords related to python closure scoping only yields various sorts of complicated PEPs and an even uglier list trick:

a lot of people might not know about the trick of using a list to box variables within a closure.

Now I know about the trick, but I wish I didn't need to know

Turbogears quirks when testing controllers that use SingleSelectField

Suppose you have a User that can be a member of a Company. In SQLObject you model it somehow like this:

    class Company(SQLObject):
        name = UnicodeCol(length=16, alternateID=True, alternateMethodName="by_name")
        display_name = UnicodeCol(length=255)

    class User(InheritableSQLObject):
        company = ForeignKey("Company", notNull=False, cascade='null')

Then you want to make a form that allows to choose what is the company of a user:

def companies():
    return [ [ -1, 'None' ] ] + [ [c.id, c.display_name] for c in Company.select() ]

class NewUserFields(WidgetsList):
    """Fields for editing general settings"""
    user_name = TextField(label="User name")
    companyID = SingleSelectField(label="Company", options=companies)

Ok. Now you want to run tests:

1. nosetests imports the controller to see if there's any initialisation code.
2. The NewUserFields class is created.
3. The SingleSelectField is created.
4. The SingleSelectField constructor tries to guess the validator and peeks at the first option.
5. This calls companies.
6. companies accesses the database.
7. The testing database has not yet been created because nosetests imported the module before giving the test code a chance to setup the test database.
8. Bang.

The solution is to add an explicit validator to disable this guessing code that is a source of so many troubles:

class NewUserFields(WidgetsList):
    """Fields for editing general settings"""
    user_name = TextField(label="User name")
    companyID = SingleSelectField(label="Company", options=companies, validator=v.Int(not_empty=True))

Turbogears form quirk

I had a great idea:

@validate(model_form)
@error_handler()
@expose(template='kid:myproject.templates.new')
def new(self, id, tg_errors=None, **kw):
    """Create new records in model"""
    if tg_errors:
        # Ask until there is still something missing
        return dict(record = defaults, form = model_form)
    else:
        # We have everything: save it
        i = Item(**kw)
        flash("Item was successfully created.")
        raise redirect("../show/%d" % i.id)

It was perfect: one simple method, simple error handling, nice helpful messages all around. Except, check boxes and select fields would not get the default values while all other fields would.

After two hours searching and cursing and tracing things into widget code, I found this bit in InputWidget.adjust_value:

# there are some input fields that when nothing is checked/selected
# instead of sending a nice name="" are totally missing from
# input_values, this little workaround let's us manage them nicely
# without interfering with other types of fields, we need this to
# keep track of their empty status otherwise if the form is going to be
# redisplayed for some errors they end up to use their defaults values
# instead of being empty since FE doesn't validate a failing Schema.
# posterity note: this is also why we need if_missing=None in
# validators.Schema, see ticket #696.

So, what is happening here is that since check boxes and option fields don't have a nice behaviour when unselected, turbogears has to work around it. So in order to detect the difference between "I selected 'None'" and "I didn't select anything", it reasons that if the input has been validated, then the user has made some selections, so it defaults to "The user selected 'None'". If the input has not been validated, then we're showing the form for the first time, then a missing value means "Use the default provided".

Since I was doing the validation all the time, this meant that Checkboxes and Select fields would never use the default values.

Hence, if you use those fields then you necessarily need two different controller methods, one to present the form and one to save it:

@expose(template='kid:myproject.templates.new')
def new(self, id, **kw):
    """Create new records in model"""
    return dict(record = defaults(), form = model_form)

@validate(model_form)
@error_handler(new)
@expose()
def savenew(self, id, **kw):
    """Create new records in model"""
    i = Item(**kw)
    flash("Item was successfully created.")
    raise redirect("../show/%d"%i.id)

If someone else stumbles on the same problem, I hope they'll find this post and they won't have to spend another two awful hours tracking it down again.

File downloads with TurboGears

In TurboGears, I had to implement a file download method, but the file required access controls so it was put in a directory not exported by Apache.

In #turbogears I've been pointed at: http://cherrypy.org/wiki/FileDownload and this is everything put together:

from cherrypy.lib.cptools import serveFile
# In cherrypy 3 it should be:
#from cherrypy.lib.static import serve_file

@expose()
def get(self, *args, **kw):
    """Access the file pointed by the given path"""
    pathname = check_auth_and_compute_pathname()
    return serveFile(pathname)

Then I needed to export some CSV:

@expose()
def getcsv(self, *args, **kw):
    """Get the data in CSV format"""
    rows = compute_data_rows()
    headers = compute_headers(rows)
    filename = compute_file_name()

    cherrypy.response.headers['Content-Type'] = "application/x-download"
    cherrypy.response.headers['Content-Disposition'] = 'attachment; filename="'+filename+'"'

    csvdata = StringIO.StringIO()
    writer = csv.writer(csvdata)
    writer.writerow(headers)
    writer.writerows(rows)

    return csvdata.getvalue()

In my case it's not an issue as I can only compute the headers after I computed all the data, but I still have to find out how to serve the CSV file while I'm generating it, instead of storing it all into a big string and returning the big string.

Passing values to turbogears widgets at display time

In turbogears, I often need to pass data to widgets at display time. Sometimes it works automatically, but sometimes, in cases like passing option lists to CheckBoxLists or number of repetitions in a RepeatingFieldSet, it doesn't.

All the examples use precomputed lists or pass simple code functions. In most of my cases, I want them computed by the controller every time.

Passing a function hasn't worked, as I did not find any obvious way to have the function know about the controller.

So I need to pass things the display() method of the widgets, but I could not work out how to pass the option list and default list for a CheckBoxList that is part of a WidgetsList in a TableForm.

On IRC came the answer, thanks to Xentac:

you should be able to...
    tableform.display(options=dict(checkboxname=[optionlist]))

And yes, it works. I can pass the default value as one of the normal form values:

    tableform.display(values=dict(checkboxname=[values]), options=dict(checkboxname=[optionlist]))

Turbogears i18n quirks

Collecting strings from .kid files

tg-admin i18n collect won't collect strings from your .kid files: you need the toolbox web interface for that.

Indentation problems in .kid files

The toolbox web interface chokes on intentation errors on your .kid files.

To see the name of the .kid file that causes the error, look at the tg-admin toolbox output in the terminal for lines like Working on app/Foo/templates/bar.kid.

What happens is that the .kid files are converted to python files, and if there are indentation glitches they end up in the python files, and python will complain.

Once you see from the tg-admin toolbox standard error what is the .kid file with the problem, edit it and try to make sure that all closing tags are at the exact indentation level as their coresponding opening tags. Even a single space would matter.

Bad i18n bug in TurboKid versions earlier than 1.0.1

faide on #turbogears also says:

It is of outmost importance that you use TurboKid 1.0.1 because it is the first version that corrects a BIG bug regarding i18n filters ...

The version below had a bug where the filters kept being added at each page load in such a way that after a few hundreds of pages you could have page loading times as long as 5 minutes!

If one has a previous version of TurboKid, one (and only one) of these is needed:

• Patch Kid with the patch at http://www.kid-templating.org/trac/ticket/203
• Patch TurboKid with the patch at http://trac.turbogears.org/ticket/1301
• Upgrade to TurboGears 1.0.2.2, which also works like a charm with python 2.5, which is a performance boost.

So, in short, all i18n users should upgrade to TurboGears 1.0.2.2 or patch TurboKid using http://trac.turbogears.org/ticket/1301.

Quirks when overriding SQLObject setters

Let's suppose you have a User that is, optionally, a member of a Company. In SQLObject you model it somehow like this:

    class Company(SQLObject):
        name = UnicodeCol(length=16, alternateID=True, alternateMethodName="by_name")
        display_name = UnicodeCol(length=255)

    class User(InheritableSQLObject):
        company = ForeignKey("Company", notNull=False, cascade='null')

Then you want to implement a user settings interface that uses a Select box to choose the company of the user.

For the Select widget to properly handle the validator for your data, you need to put a number in the first option. As my first option, I want to have the "None" entry, so I decided to use -1 to mean "None".

Now, to make it all blend nicely, I overrode the company setter to accept -1 and silently convert it to a None:

    class User(InheritableSQLObject):
        company = ForeignKey("Company", notNull=False, cascade='null')

        def _set_company(self, id):
            "Set the company id, using None if -1 is given"
            if id == -1: id = None
            self._SO_set_company(id)

In the controller, after parsing and validating all the various keyword arguments, I do something like this:

            user.set(**kw)

Now, the overridden method didn't get called.

After some investigation, and with the help of NandoFlorestan on IRC, we figured out the following things:

1. That method needs to be rewritten as _set_companyID:

           def _set_companyID(self, id):
               "Set the company id, using None if -1 is given"
               if id == -1: id = None
               self._SO_set_companyID(id)
   
   

2. Methods overridden in that way are alsop called by user.set(**kw), but not by the User(**kw) constructor, so using, for example, a similar override to transparently encrypt passwords would give you plaintext passwords for new users and encrypted passwords after they changed it.

Passing values to turbogears widgets at display time (the general case)

Last time I dug this up I was not clear enough in documenting my findings, so I had to find them again. Here is the second attempt.

In Turbogears, in order to pass parameters to arbitrary widgets in a compound widget, the syntax is:

form.display(PARAMNAME=dict(WIDGETNAME=VALUE))

And if you have more complex nested widgets and would like to know what goes on, this monkey patch is good for inspecting the params lookup functions:

import turbogears.widgets.forms
old_rpbp = turbogears.widgets.forms.retrieve_params_by_path
def inspect_rpbp(params, path):
    print "RPBP", repr(params), repr(path)
    res = old_rpbp(params, path)
    print "RPBP RES", res
    return res
turbogears.widgets.forms.retrieve_params_by_path = inspect_rpbp

The code for the lookup itself is, as the name suggests, in the retrieve_params_by_path function in the file widgets/forms.py in the Turbogears source code.

Linking to self in turbogears

I want to put in my master.kid some icons that allow to change the current language for the session.

First, all user-accessible methods need to handle a 'language' parameter:

@expose(template="myapp.templates.foobar")
def index(self, someparam, **kw):
    if 'language' in kw: turbogears.i18n.set_session_locale(kw['language'])

Then, we need a way to edit the current URL so that we can generate modified links to self that preserve the existing path_info and query parameters. In your main controller, add:

def linkself(**kw):
    params = {}
    params.update(cherrypy.request.params)
    params.update(kw)
    url = cherrypy.request.browser_url.split('?', 1)[0]
    return url + '?' + '&'.join(['='.join(x) for x in params.iteritems()])

def add_custom_stdvars(vars):
    return vars.update({"linkself": linkself})

turbogears.view.variable_providers.append(add_custom_stdvars)

(see the turbogears stdvars documentation and the cherrypy request documentation (cherrypy 2 documentation at the bottom of the page))

And finally, in master.kid:

<div id="footer">
  <div id="langselector">
    <span class="language">
      <a href="${tg.linkself(language='it_IT')}">
        <img src="${tg.url('/static/images/it.png')}"/>
      </a>
    </span>

    <span class="language">
      <