How to read the Freerunner's accelerometers

This code has been take from moko_eightball by Jakob Westhoff: it just continuously prints the value of the three accelerometers.

#include <stdio.h>
#include <stdint.h>

void processInputEvents(FILE* in)
{
    int x = 0, y = 0, z = 0;
    while (1)
    {
        char padding[16];
        uint16_t type, code;
        int32_t value;

        // Skip the timestamp
        fread(padding, 1, 8, in);

        // Read the type
        fread(&type, 1, 2, in);

        // Read the code
        fread(&code, 1, 2, in);

        // Read the value
        fread(&value, 1, 4, in);

        switch( type )
        {
            case 0:
                switch( code )
                {
                    case 0:
                        fprintf(stdout, "x%d y%d z%d\n", x, y, z);
                        break;
                    default:
                        //warning( "Unknown code ( 0x%02x ) for type 0x%02x\n", code, type );
                        break;
                }
                break;
            case 2:
                switch ( code )
                {
                    case 0:
                        // Update to the new value
                        x = value;
                        break;
                    case 1:
                        // Update to the new value
                        y = value;
                        break;
                    case 2:
                        // Update to the new value
                        z = value;
                        break;
                    default:
                        //warning( "Unknown code ( 0x%02x ) for type 0x%02x\n", code, type );
                        break;
                }
                break;

            default:
                //warning( "Unknown type ( 0x%02x ) in accelerometer input stream\n", type );
                break;
        }


    }
}


int main()
{
    FILE* in = fopen("/dev/input/event2", "r");
    processInputEvents(in);
    fclose(in);
    return 0;
}

Polysms

Here is my first software designed for the FreeRunner: polysms. It's a commandline tool: you pass it a polygen grammar name and a phone number, and it will send a SMS to that phone number using the polygen output for that grammar as the SMS text:

# polyrun manager 0012345678

And here is the code, that works on the http://www.freesmartphone.org dbus framework:

#!/usr/bin/python

# (C) 2008 Enrico Zini
# Most bits of this are stripped from zhone, which is:
# (C) 2007 Johannes 'Josch' Schauer
# (C) 2008 Michael 'Mickey' Lauer <mlauer@vanille-media.de>
# (C) 2008 Jan 'Shoragan' Luebbe
# (C) 2008 Daniel 'Alphaone' Willmann
# (C) 2008 Openmoko, Inc.
# GPLv2 or later

from dbus import SystemBus, Interface
from dbus.exceptions import DBusException
import logging
logger = logging.getLogger( __name__ )

from dbus.mainloop.glib import DBusGMainLoop
DBusGMainLoop(set_as_default=True)

import gobject
import sys
from subprocess import Popen, PIPE


class Phone:
    def tryGetProxy( self, busname, objname ):
        try:
            return self.bus.get_object( busname, objname )
        except DBusException, e:
            logger.warning( "could not create proxy for %s:%s" % ( busname, objname ) )

    def __init__(self):
        try:
            self.bus = SystemBus()
        except DBusException, e:
            logger.error( "could not connect to dbus_object system bus: %s" % e )
            return False

        # Phone
        self.gsm_device_obj = self.tryGetProxy( 'org.freesmartphone.ogsmd', '/org/freesmartphone/GSM/Device' )

        if ( self.gsm_device_obj is not None ):
            self.gsm_device_iface = Interface(self.gsm_device_obj, 'org.freesmartphone.GSM.Device')
            self.gsm_sim_iface = Interface(self.gsm_device_obj, 'org.freesmartphone.GSM.SIM')
            self.gsm_network_iface = Interface(self.gsm_device_obj, 'org.freesmartphone.GSM.Network')
            self.gsm_call_iface = Interface(self.gsm_device_obj, 'org.freesmartphone.GSM.Call')
            self.gsm_test_iface = Interface(self.gsm_device_obj, 'org.freesmartphone.GSM.Test')

        # Main loop
        self.loop = gobject.MainLoop()

    def send(self, number, message):
        def onSent():
            print "SENT"
            self.loop.quit()
        def onStore(index):
            print "STORED AS", index
            self.gsm_sim_iface.SendStoredMessage(
                index,
                reply_handler=onSent,
                error_handler=self.onError
            )
        self.gsm_sim_iface.StoreMessage(
            number, message,
            reply_handler=onStore,
            error_handler=self.onError
        )

    def onError(self, result):
        print "ERROR", result

    def mainloop(self):
        self.loop.run()

if len(sys.argv) != 3:
    print >>sys.stderr, "Usage: %s grammarname phonenumber"
    sys.exit(1)

message = Popen(["/usr/bin/polyrun", sys.argv[1]], stdout=PIPE).communicate()[0]
number = sys.argv[2]

print "Sending to %s:" % number
print message

phone = Phone()
phone.send(number, message)
phone.mainloop()

Docking the FreeRunner into the laptop

Earlier in the day, I wrote:

So yes, my laptop can now be turned into a phone charger with networking, DNS and apt cache services. I shall look into hooking that script into dbus to have it run automatically when the phone is plugged and unplugged.

It's not dbus, it's udev, and I've managed to do it.

It's an udev rule:

# cat /etc/udev/rules.d/z60_openmoko_net.rules
ACTION=="add", SUBSYSTEM=="net", ATTRS{idVendor}=="1457", ATTRS{idProduct}=="5122", RUN+="/root/bin/share-openmoko"
ACTION=="remove", SUBSYSTEM=="net", INTERFACE="usb0", RUN+="/root/bin/share-openmoko"

And a script:

# cat bin/share-openmoko 
#!/bin/sh

if [ "$ACTION" == "add" ]
then
    ACTION=start
fi

if [ "$ACTION" == "remove" ]
then
    ACTION=stop
fi

INTERFACE=${INTERFACE:-"usb0"}
ACTION=${ACTION:-"$1"}

case "$ACTION" in
    start)
        logger -t openmoko "Connected, setting up network"
        iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE -s 192.168.0.200/29
        ifconfig usb0 192.168.0.200 netmask 255.255.255.248
        /etc/init.d/dnsmasq start
        echo 1 > /proc/sys/net/ipv4/ip_forward
        ;;
    stop)
        logger -t openmoko "Disconnected, bringing down network"
        echo 0 > /proc/sys/net/ipv4/ip_forward
        /etc/init.d/dnsmasq stop
        iptables -t nat -F POSTROUTING
        ifconfig usb0 down
        ;;
esac

The script has extra cruft that makes it double as a udev script and as a init.d style script, just because I didn't feel like abandoning the init.d style interface. However, udev handles it perfectly, so there's probably no use at all for the start/stop part.

This means that the interface and all supporting services will be brought up and down when the phone is connected/disconnected, and also when the phone is suspended/resumed. Of course, more fancy things can be plugged into the script, like syncing PIM info, file systems, turning on applets in panels, mounting phone file systems and whatnot.

Unpacking the new FreeRunner

I got myself a FreeRunner. Here are some notes from the first few days:

Available operating systems

The FreeRunner comes with the Om 2007.2 distribution: it works for basic phone things, and it has an opkg package manager that you can use to install all sort of extra software. Only issue: the SMS application was rather unstable with my SIM card.

I then tried Om 2008.8, which is a major new redesign, partly based on Qtopia. It's definitely more advanced on how it looks, with smooth animations all around, but I did not manage to get the GSM part to work, because I did not manage to get it to ask for a PIN.

I then tried plain Qtopia and I got what looks and feels like a properly working mobile phone. The only issue that it has is that I did not manage to make it suspend, so battery life is much shorter than it could be. Apparently, this should improve as kernel 2.6.26 reaches the phone. I kept Qtopia installed in the phone flash as a "stable" phone system.

And finally, Debian. Debian is based on the freesmartphone.org software stack, which is an attempt to create a UI agnostic DBUS frontend to the hardware that supports multiple applications running on top of it. It is a young project, but it can already drive a mobile phone in a useful way. It has a demo interface that is basically a showcase of what is implemented in the DBUS frontend, but does most of the basic things you need from a mobile phone, including taking and making phone calls. And then it has Debian behind, all of it. I need to buy a bigger microsd card.

Tips and tricks

Flashing things

1. apt-get install dfu-util
2. Start the Freerunner bios/bootloader, by holding down Power and then pressing AUX (standard bootloader), or by holding down AUX and then pressing Power (factory, unbrickable, read only fail safe bootloader).
3. dfu-util -l shows you a list of devices it can access. If you see more than once, you need to specify in all dfu-util commands which one you want, using -d USBID. In my case, I have to always use dfu-util -d 0x1d50:0x5119
4. To flash the kernel: dfu-util -a kernel -R -D /path/to/uImage
5. To flash the root file system: dfu-util -a rootfs -R -D rootfs_filename.jffs2
6. To flash the bootloader: dfu-util -a u-boot -R -D uboot_filename.bin
7. To flash the u-boot configuration: dfu-util -a u-boot_env -D env.new

You can also download all of these things from the FreeRunner by using -U instead of -D.

Networking via USB

All of the distributions I tried, by default configure the USB as a gadget with ethernet over usb. You can form a lan with it using the cdc_ether module, and you will find your phone preconfigured as 192.168.0.202 expecting to find a gateway at 192.168.0.200.

I made myself this script to start and stop networking with the phone:

#!/bin/sh

case "$1" in
    start)
        iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE -s 192.168.0.200/29
        ifconfig usb0 192.168.0.200 netmask 255.255.255.248
        /etc/init.d/dnsmasq start
        echo 1 > /proc/sys/net/ipv4/ip_forward
        ;;
    stop)
        echo 0 > /proc/sys/net/ipv4/ip_forward
        /etc/init.d/dnsmasq stop
        iptables -t nat -F POSTROUTING
        ifconfig usb0 down
        ;;
esac

Besides doing masquerading, it also brings up dnsmasq so that the phone can always find a DNS together with the router.

Another useful trick is to configure the phone to share the approx cache with the laptop:

deb http://192.168.0.200:9999/debian unstable main
deb http://192.168.0.200:9999/debian experimental main
deb http://pkg-fso.alioth.debian.org/debian unstable main

So yes, my laptop can now be turned into a phone charger with networking, DNS and apt cache services. I shall look into hooking that script into dbus to have it run automatically when the phone is plugged and unplugged.

Changing the ringtone in Debian

In case you don't like the default ringtone, this is how to change it:

1. vi /usr/share/python-support/fso-frameworkd/framework/subsystems/oeventd/receiver.py
2. look for def _play( self ):
3. change the codec in decoder = gst.element_factory_make, if needed. The list of available plugins is on the gstreamer website.
4. change the path in filesrc.set_property
5. /etc/init.d/fso-frameworkd restart and maybe /etc/init.d/zhone-session restart

Yes, the ringtone is currently hardcoded, but it does say it's a prototype after all. Or, if you prefer, it's fully configurable and the configuration can be found in /usr/share/python-support/fso-frameworkd.

Configuring the bootloader

You can connect to the u-boot bootloader via a serial terminal on /dev/ttyACM0. Type help and you will find that it can do a lot of things. The OpenMoko wiki has pages on the bootloader itself, its commands and the environment.

The environment is the configuration of the bootloader, similar somehow to /boot/grub/menu.lst. Unlike grub, you can edit the environment from within the bootloader and then save it using the saveenv command.

What I did:

1. A boot entry for Debian, with the kernel on an ext2 partition instead of fat: setenv menu_3 Boot from microSD (FAT+ext2): setenv bootargs \${bootargs_base} rootfstype=ext2 root=/dev/mmcblk0p2 rootdelay=5 \${mtdparts} ro\; mmcinit\; fatload mmc 1 0x32000000 \${sd_image_name}\; bootm 0x32000000
2. Stay in the bootloader until a choice has been made: setenv bootdelay -1
3. Don't power down the bootloader when idle: setenv boot_menu_timeout 99999
4. Always show the menu at boot: setenv stop_in_menu yes
5. saveenv

So now my phone dual boots, and I can choose if I want a more reliable phone now (Qtopia) or if I want to play with my future phone (Debian).

Configuring ssh for two host keys on the same host

Minor issue, but annoying: since both QTopia and Debian show up on 192.168.0.202, ssh will complain about changed host keys. Here is how to configure ssh to avoid the problem (in ~/.ssh/config):

Host debian
HostName 192.168.0.202
User root
HostKeyAlias debian

Host qtopia
HostName 192.168.0.202
User root
HostKeyAlias qtopia

apt-xapian-index: dynamically generated tag clouds

About apt-xapian-index, I have already posted:

• an introduction of apt-xapian-index;
• an example of how to query it;
• a way to add simple result filters to the query;
• a way to suggest keywords and tags to use to improve the query.
• a way to search for similar packages.
• a way to implement an adaptive cutoff on result quality.
• a smart way of querying tags
• how to implement search as you type

Today I'll show how to create tag clouds. Not only that, but I'll show how to implement tag clouds that change as the user types a query.

This example uses python-gtk, and has been created together with Matteo Zandi.

Generating a tag cloud out of any Xapian query is simple, and it is just a matter of presenting into a tag cloud the information that you get with the technique shown in a smart way of querying tags: you get the tags related to the query, and you lay out their names with a font size proportional to their Xapian rank.

For the presentation, we can load pretty names from the Debtags vocabulary in /var/lib/debtags/vocabulary:

from debian_bundle import deb822

# Facet name -> Short description
facets = dict()
# Tag name -> Short description
tags = dict()
for p in deb822.Deb822.iter_paragraphs(open("/var/lib/debtags/vocabulary", "r")):
    if "Description" not in p: continue
    desc = p["Description"].split("\n", 1)[0]
    if "Tag" in p:
        tags[p["Tag"]] = desc
    elif "Facet" in p:
        facets[p["Facet"]] = desc

The query then goes on as usual, and when we get the tags from the eset we also record their score and normalise it between 0 and 1. I found that computing the logarithm of scores helps to avoid having a tag cloud with a few huge tags and a lot of tiny tiny tags:

class Filter(xapian.ExpandDecider):
    def __call__(self, term):
        return term[:2] == "XT"

def format(k):
    if k in tags:
        facet = k.split("::", 1)[0]
        if facet in facets:
            return "<i>%s: %s</i>" % (facets[facet], tags[k])
        else:
            return "<i>%s</i>" % tags[k]
    else:
        return k

taglist = []
maxscore = None
for res in enquire.get_eset(15, rset, Filter()):
    # Normalise the score in the interval [0, 1]
    weight = math.log(res.weight)
    if maxscore == None: maxscore = weight
    tag = res.term[2:]
    taglist.append(
        (tag, format(tag), float(weight) / maxscore)
    )
taglist.sort(key=lambda x:x[0])

Finally, you mark up a gtkhtml2.Document to display in a gtkhtml2 widget:

def mark_text_up(result_list):
    # 0-100 score, key (facet::tag), description
    document = gtkhtml2.Document()
    document.clear()
    document.open_stream("text/html")
    document.write_stream("""<html><head>
<style type="text/css">
a { text-decoration: none; color: black; }
</style>
</head><body>""")
    for tag, desc, score in result_list:
        document.write_stream('<a href="%s" style="font-size: %d%%">%s</a> ' % (tag, score*150, desc))
    document.write_stream("</body></html>")
    document.close_stream()
    return document

That's it, try it out.

You can use the git web interface to get to the full source code and the module it uses.

Handling concurrent update-apt-xapian-index runs

So, update-apt-xapian-index takes a bit of a long time to run. When you install the package, if the index isn't there then postinst will rebuild it in background, good.

Now, suppose you have an advanced package manager that after installing apt-xapian-index, detects that it's installed, detects that there is no database and runs an update with a nice GUI feedback. If this happened while the update is running in the background, the usual thing to do would be to abort the update; however, the package manager wouldn't be able to know when the indexing has finished and it can enable the advanced features.

Let's not do the usual thing, then:

# `progress' is the object that does progress reporting.  Commandline options
# can choose a silent one, a machine readable one or a user readable one.

# Lock the session so that we prevent concurrent updates
lockfd = os.open(XAPIANDBLOCK, os.O_RDWR | os.O_CREAT)
lockpyfd = os.fdopen(lockfd)
try:
    fcntl.lockf(lockpyfd, fcntl.LOCK_EX | fcntl.LOCK_NB)

    # `ServerProgress` is a proxy progress reporter that listens for
    # connections on a Unix socket, and broadcasts progress status updates both
    # to the local progress reporter and to whoever connects to the socket.
    progress = ServerProgress(progress)
except IOError, e:
    if e.errno == errno.EACCES or e.errno == errno.EAGAIN:
        progress.notice("Another update is already running: showing its progress.")
        # `ClientProgress' is another proxy class that connects to the socket
    # forwards progress updates to whatever the user has chosen as the
    # local progress indicator.
        childProgress = ClientProgress(progress)
        childProgress.loop()
        sys.exit(0)
    else:
        raise

What happens here is that if the upload is in progress, the second update-apt-xapian-index will not be able to lock a lockfile. But instead of exiting, it connects to the other indexer via a unix socket, and shows the progress just as if it were the one doing the indexing.

So, now, suppose I run update-apt-xapian-index:

# update-apt-xapian-index
Reading Debtags database: done.  
Rebuilding Xapian index... 20%

And suppose I run another one:

# update-apt-xapian-index
Another update is already running: showing its progress.
Rebuilding Xapian index... 20%

And suppose I run yet another one:

# update-apt-xapian-index --batch-mode
notice: Another update is already running: showing its progress.
begin: Rebuilding Xapian index
progress: 19/100
progress: 20/100

And suppose I run yet another one (running with -q shows nothing, but it will wait until the master indexer has finished):

# update-apt-xapian-index -q

This way, the advanced package manager (adept 3 does this) will show a progress indicator and correctly let the user know when indexing has finished... only, if an update is already running, the progress indicator will not start at 0% because the update was already in progress, and the user will think: "wow, I've been lucky!".

The server side of the socket only uses nonblocking I/O, so there's no risk of slave indexers causing the master indexer to hang.

It's been simple to implement, and brillant. It is in apt-xapian-index 0.15, which I have just uploaded to sid. Shame it's a bit late for lenny.

Introducing the Humongous Merged Packages File From Hell

Suppose you want to build some service that requires information about all packages in all architectures. Like, for example, the debtags web tagging interface.

If that is the case, then you may be interested in the Humongous Merged Packages File From Hell.

It contains a merge of all Packages and Source files of etch, lenny, sid and experimental, for main, contrib and non-free, in all architectures.

The merge is done according to completely arbitrary criteria of mine:

• One record per (package, version) couple found; multiple record per (package, version) are merged together
• Size and Installed-Size are the average of all values found
• MD5sum, SHA1, SHA256, Filename, Files, Directory, Checksums-Sha1, Checksums-Sha256, Binary are thrown away
• Information (such as Homepage, Vcs* fields, Uploaders) found in the Sources file is repeated for every binary package
• All other packages get as value a list of all the values found, with duplicates removed

It's there in case anyone other than me may find it useful.