Index of categories

Entries to also be published on truelite.it

Free as in Facebook

Yesterday we were in an airport. We tried to connect to the airport "free" wifi. It had a captive portal that asked for a lot of personal information before one could maybe get on the internet, and we gave up. Bologna Airport, no matter what they do to pretend that they like you, it's always clear that they don't.

I looked at the captive portal screen and I said: «ah yes, "free" wifi. Free as in Facebook».

We figured that we had an expression that will want to be reused.

Posted Mon Mar 9 10:58:49 2015 Tags:

Setting up Akonadi

Now that I have a CalDAV server that syncs with my phone I would like to use it from my desktop.

It looks like akonadi is able to sync with CalDAV servers, so I'm giving it a try.

First thing first is to give a meaning to the arbitrary name of this thing. Wikipedia says it is the oracle goddess of justice in Ghana. That still does not hint at all at personal information servers, but seems quite nice. Ok. I gave up with software having purpose-related names ages ago.

# apt-get install akonadi-server akonadi-backend-postgresql

Akonadi wants a SQL database as a backend. By default it uses MySQL, but I had enough of MySQL ages ago.

I tried SQLite but the performance with it is terrible. Terrible as in, it takes 2 minutes between adding a calendar entry and having it show up in the calendar. I'm fascinated by how Akonadi manages to use SQLite so badly, but since I currently just want to get a job done, next in line is PostgreSQL:

# su - postgres
$ createuser enrico
$ psql postgres
postgres=# alter user enrico createdb;

Then as enrico:

$ createdb akonadi-enrico
$ cat <<EOT > ~/.config/akonadi/akonadiserverrc
[%General]
Driver=QPSQL

[QPSQL]
Name=akonadi-enrico
StartServer=false
Host=
Options=
ServerPath=
InitDbPath=

I can now use kontact to connect Akonadi to my CalDAV server and it works nicely, both with calendar and with addressbook entries.

KDE has at least two clients for Akonadi: Kontact, which is a kitchen sink application similar to Evolution, and KOrganizer, which is just the calendar and scheduling component of Kontact.

Both work decently, and KOrganizer has a pretty decent startup time. I now have a usable desktop PIM application that is synced with my phone. W00T!

Next step is to port my swift little calendar display tool to use Akonadi as a back-end.

Posted Tue Feb 17 15:34:55 2015 Tags:

seat-inspect

Four months ago I wrote this somewhere:

Seeing a DD saying "this new dbus stuff scares me" would make most debian users scared. Seeing a DD who has an idea of what is going on, and who can explain it, would be an interesting and exciting experience.

So, let's be exemplary, competent and patient. Or at least, competent. Some may like or not like the changes, but do we all understand what is going on? Will we all be able to support our friends and customers running jessie?

I confess that although I understand the need for it, I don't feel competent enough to support systemd-based machines right now.

So, are we maybe in need of help, cheat sheets, arsenals of one-liners, diagnostic tools?

Maybe a round of posts on -planet like "one debian package a day" but with new features that jessie will have, and how to understand them and take advantage of them?

That was four months ago. In the meantime, I did some work, and it got better for me.

Yesterday, however, I've seen an experienced Linux person frustrated because the shutdown function of the desktop was doing nothing whatsoever. Today I found John Goerzen's post on planet.

I felt like some more diagnostic tools were needed, so I spent the day making seat-inspect.

seat-inspect tries to make the status of the login/seat system visible, to help with understanding and troubleshooting.

The intent of running the code is to have an overview of the system status, both to see what the new facilities are about, and to figure out if there is something out of place.

The intent of reading the code is to have an idea of how to use these facilities: the code has been written to be straightforward and is annotated with relevant bits from the logind API documentation.

seat-inspect is not a finished tool, but a starting point. I put it on github hoping that people will fork it and add their own extra sanity checks and warnings, so that it can grow into a standard thing to run if a system acts weird.

As it is now, it should be able to issue warnings if some bits are missing for network-manager or shutdown functions to work correctly. I haven't really tested that, though, because I don't have a system at hand where they are currently not working fine.

Another nice thing of it is that when running seat-inspect -v you get a dump of what logind/consolekit think about your system. I found it an interesting way to explore the new functionalities that we recently grew. The same can be done, and in more details, with loginctl calls, but I lacked a summary.

After writing this I feel a bit more competent, probably enough to sit at somebody's computer and poke into loginctl bits. I highly recommend the experience.

Posted Tue Feb 10 18:06:43 2015 Tags:

Playing with python, terminfo and command output

I am experimenting with showing progress on the terminal for a subcommand that is being run, showing what is happening without scrolling away the output of the main program, and I came out with this little toy. It shows the last X lines of a subcommand output, then gets rid of everything after the command has ended.

Usability-wise, it feels like a tease to me: it looks like I'm being shown all sorts of information then they are taken away from me before I managed to make sense of them. However, I find it cute enough to share:

#!/usr/bin/env python3
#coding: utf-8
# Copyright 2015 Enrico Zini <enrico@enricozini.org>.  Licensed under the terms
# of the GNU General Public License, version 2 or any later version.

import argparse
import fcntl
import select
import curses
import contextlib
import subprocess
import os
import sys
import collections
import shlex
import shutil
import logging

def stream_output(proc):
    """
    Take a subprocess.Popen object and generate its output, line by line,
    annotated with "stdout" or "stderr". At process termination it generates
    one last element: ("result", return_code) with the return code of the
    process.
    """
    fds = [proc.stdout, proc.stderr]
    bufs = [b"", b""]
    types = ["stdout", "stderr"]
    # Set both pipes as non-blocking
    for fd in fds:
        fcntl.fcntl(fd, fcntl.F_SETFL, os.O_NONBLOCK)
    # Multiplex stdout and stderr with different prefixes
    while len(fds) > 0:
        s = select.select(fds, (), ())
        for fd in s[0]:
            idx = fds.index(fd)
            buf = fd.read()
            if len(buf) == 0:
                fds.pop(idx)
                if len(bufs[idx]) != 0:
                    yield types[idx], bufs.pop(idx)
                types.pop(idx)
            else:
                bufs[idx] += buf
                lines = bufs[idx].split(b"\n")
                bufs[idx] = lines.pop()
                for l in lines:
                    yield types[idx], l
    res = proc.wait()
    yield "result", res

@contextlib.contextmanager
def miniscreen(has_fancyterm, name, maxlines=3, silent=False):
    """
    Show the output of a process scrolling in a portion of the screen.

    has_fancyterm: true if the terminal supports fancy features; if false, just
    write lines to standard output

    name: name of the process being run, to use as a header

    maxlines: maximum height of the miniscreen

    silent: do nothing whatsoever, used to disable this without needing to
            change the code structure

    Usage:
        with miniscreen(True, "my process", 5) as print_line:
            for i in range(10):
                print_line(("stdout", "stderr")[i % 2], "Line #{}".format(i))
    """
    if not silent and has_fancyterm:
        # Discover all the terminal control sequences that we need
        output_normal = str(curses.tigetstr("sgr0"), "ascii")
        output_up = str(curses.tigetstr("cuu1"), "ascii")
        output_clreol = str(curses.tigetstr("el"), "ascii")
        cols, lines = shutil.get_terminal_size()
        output_width = cols

        fg_color = (curses.tigetstr("setaf") or
                    curses.tigetstr("setf") or "")
        sys.stdout.write(str(curses.tparm(fg_color, 6), "ascii"))

        output_lines = collections.deque(maxlen=maxlines)

        def print_lines():
            """
            Print the lines in our buffer, then move back to the beginning
            """
            sys.stdout.write("{} progress:".format(name))
            sys.stdout.write(output_clreol)
            for msg in output_lines:
                sys.stdout.write("\n")
                sys.stdout.write(msg)
                sys.stdout.write(output_clreol)
            sys.stdout.write(output_up * len(output_lines))
            sys.stdout.write("\r")

        try:
            print_lines()

            def _progress_line(type, line):
                """
                Print a new line to the miniscreen
                """
                # Add the new line to our output buffer
                msg = "{} {}".format("." if type == "stdout" else "!", line)
                if len(msg) > output_width - 4:
                    msg = msg[:output_width - 4] + "..."
                output_lines.append(msg)
                # Update the miniscreen
                print_lines()

            yield _progress_line

            # Clear the miniscreen by filling our ring buffer with empty lines
            # then printing them out
            for i in range(maxlines):
                output_lines.append("")
            print_lines()
        finally:
            sys.stdout.write(output_normal)
    elif not silent:
        def _progress_line(type, line):
            print("{}: {}".format(type, line))
        yield _progress_line
    else:
        def _progress_line(type, line):
            pass
        yield _progress_line

def run_command_fancy(name, cmd, env=None, logfd=None, fancy=True, debug=False):
    quoted_cmd = " ".join(shlex.quote(x) for x in cmd)
    log.info("%s running command %s", name, quoted_cmd)
    if logfd: print("runcmd:", quoted_cmd, file=logfd)

    # Run the script itself on an empty environment, so that what was
    # documented is exactly what was run
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)

    with miniscreen(fancy, name, silent=debug) as progress:
        stderr = []
        for type, val in stream_output(proc):
            if type == "stdout":
                val = val.decode("utf-8")
                if logfd: print("stdout:", val, file=logfd)
                log.debug("%s stdout: %s", name, val)
                progress(type, val)
            elif type == "stderr":
                val = val.decode("utf-8")
                if logfd: print("stderr:", val, file=logfd)
                stderr.append(val)
                log.debug("%s stderr: %s", name, val)
                progress(type, val)
            elif type == "result":
                if logfd: print("retval:", val, file=logfd)
                log.debug("%s retval: %d", name, val)
                retval = val

    if retval != 0:
        lastlines = min(len(stderr), 5)
        log.error("%s exited with code %s", name, retval)
        log.error("Last %d lines of standard error:", lastlines)
        for line in stderr[-lastlines:]:
            log.error("%s: %s", name, line)

    return retval


parser = argparse.ArgumentParser(description="run a command showing only a portion of its output")
parser.add_argument("--logfile", action="store", help="specify a file where the full execution log will be written")
parser.add_argument("--debug", action="store_true", help="debugging output on the terminal")
parser.add_argument("--verbose", action="store_true", help="verbose output on the terminal")
parser.add_argument("command", nargs="*", help="command to run")
args = parser.parse_args()

if args.debug:
    loglevel = logging.DEBUG
elif args.verbose:
    loglevel = logging.INFO
else:
    loglevel = logging.WARN
logging.basicConfig(level=loglevel, stream=sys.stderr)
log = logging.getLogger()

fancy = False
if not args.debug and sys.stdout.isatty():
    curses.setupterm()
    if curses.tigetnum("colors") > 0:
        fancy = True

if args.logfile:
    logfd = open("output.log", "wt")
else:
    logfd = None

retval = run_command_fancy("miniscreen example", args.command, logfd=logfd)

sys.exit(retval)
Posted Wed Jan 21 11:13:31 2015 Tags:

Upgrade Cyanogenmod with an encrypted phone

Cyanogenmod found an update, it downloaded it, then it rebooted to install it and nothing happened. It turns out that the update procedure cannot work if the zip file to install is in encrypted media, so a workaround is to move the zip into unencrypted external storage.

As far as I know, my Nexus 4 has no unencrypted external storage.

This is how I managed to upgrade it, I write it here so I can find it next time:

  1. enable USB debugging
  2. adb pull /cmupdater/cm-11-20141115-SNAPSHOT-M12-mako.zip
  3. adb reboot recovery
  4. choose "install zip from sideload"
  5. adb sideload cm-11-20141115-SNAPSHOT-M12-mako.zip
Posted Fri Dec 19 10:21:29 2014 Tags:

Radicale and DAVDroid

radicale and DAVdroid appeal to me. Let's try to make the whole thing work.

A self-signed SSL certificate

Generating the certificate:

    openssl req -nodes -x509 -newkey rsa:2048 -keyout cal-key.pem -out cal-cert.pem -days 3650
    [...]
    Country Name (2 letter code) [AU]:IT
    State or Province Name (full name) [Some-State]:Bologna
    Locality Name (eg, city) []:
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:enricozini.org
    Organizational Unit Name (eg, section) []:
    Common Name (e.g. server FQDN or YOUR name) []:cal.enricozini.org
    Email Address []:postmaster@enricozini.org

Installing it on my phone:

    openssl x509 -in cal-cert.pem -outform DER -out cal-cert.crt
    adb push cal-cert.crt /mnt/sdcard/
    enrico --follow-instructions http://davdroid.bitfire.at/faq/entry/importing-a-certificate

Installing radicale in my VPS

An updated radicale package, with this patch to make it work with DAVDroid:

    apt-get source radicale
    # I reviewed 063f7de7a2c7c50de5fe3f8382358f9a1124fbb6
    git clone https://github.com/Kozea/Radicale.git
    Move the python code from git to the Debian source
    dch -v 0.10~enrico  "Pulled in the not yet released 0.10 work from upstream"
    debuild -us -uc -rfakeroot

Install the package:

    # dpkg -i python-radicale_0.10~enrico0-1_all.deb
    # dpkg -i radicale_0.10~enrico0-1_all.deb

Create a system user to run it:

    # adduser --system --disabled-password radicale

Configure it for mod_wsgi with auth done by Apache:

    # For brevity, this is my config file with comments removed

    [storage]
    # Storage backend
    # Value: filesystem | multifilesystem | database | custom
    type = filesystem

    # Folder for storing local collections, created if not present
    filesystem_folder = /var/lib/radicale/collections

    [logging]
    config = /etc/radicale/logging

Create the wsgi file to run it:

    # mkdir /srv/radicale
    # cat <<EOT > /srv/radicale/radicale.wsgi
    import radicale
    radicale.log.start()
    application = radicale.Application()
    EOT
    # chown radicale.radicale /srv/radicale/radicale.wsgi
    # chmod 0755 /srv/radicale/radicale.wsgi

Make radicale commit to git

    # apt-get install python-dulwich
    # cd /var/lib/radicale/collections
    # git init
    # chown radicale.radicale -R /var/lib/radicale/collections/.git

Apache configuration

Add a new site to apache:

    $ cat /etc/apache2/sites-available/cal.conf
    # For brevity, this is my config file with comments removed
    <IfModule mod_ssl.c>
    <VirtualHost *:443>
            ServerName cal.enricozini.org
            ServerAdmin enrico@enricozini.org

            Alias /robots.txt /srv/radicale/robots.txt
            Alias /favicon.ico /srv/radicale/favicon.ico

            WSGIDaemonProcess radicale user=radicale group=radicale threads=1 umask=0027 display-name=%{GROUP}
            WSGIProcessGroup radicale
            WSGIScriptAlias / /srv/radicale/radicale.wsgi

            <Directory /srv/radicale>
                    # WSGIProcessGroup radicale
                    # WSGIApplicationGroup radicale
                    # WSGIPassAuthorization On
                    AllowOverride None
                    Require all granted
            </Directory>

            <Location />
                    AuthType basic
                    AuthName "Enrico's Calendar"
                    AuthBasicProvider file
                    AuthUserFile /usr/local/etc/radicale/htpasswd
                    Require user enrico
            </Location>

            ErrorLog{APACHE_LOG_DIR}/cal-enricozini-org-error.log
            LogLevel warn

            CustomLog{APACHE_LOG_DIR}/cal-enricozini-org-access.log combined

            SSLEngine on
            SSLCertificateFile    /etc/ssl/certs/cal.pem
            SSLCertificateKeyFile /etc/ssl/private/cal.key
    </VirtualHost>
    </IfModule>

Then enable it:

    # a2ensite cal.conf
    # service apache2 reload

Create collections

DAVdroid seems to want to see existing collections on the server, so we create them:

    $ apt-get install cadaver
    $ cat <<EOT > /tmp/empty.ics
    BEGIN:VCALENDAR
    VERSION:2.0
    END:VCALENDAR
    EOT
    $ cat <<EOT > /tmp/empty.vcf
    BEGIN:VCARD
    VERSION:2.1
    END:VCARD
    EOT
    $ cadaver https://cal.enricozini.org
    WARNING: Untrusted server certificate presented for `cal.enricozini.org':
    [...]
    Do you wish to accept the certificate? (y/n) y
    Authentication required for Enrico's Calendar on server `cal.enricozini.org':
    Username: enrico
    Password: ****
    dav:/> cd enrico/contacts.vcf/
    dav:/> put /tmp/empty.vcf
    dav:/> cd ../calendar.ics/
    dav:/> put /tmp/empty.ics
    dav:/enrico/calendar.ics/> ^D
    Connection to `cal.enricozini.org' closed.

DAVdroid configuration

  1. Add a new DAVdroid sync account
  2. Use server/username configuration
  3. For server, use https:////
  4. Add username and password

It should work.

Related links

Posted Tue Dec 9 16:35:50 2014 Tags:

Alternate rescue boot entry with systemd

Since systemd version 215, adding systemd.debug-shell to the kernel command line activates the debug shell on tty9 alongside the normal boot. I like the idea of that, and I'd like to have it in my standard 'rescue' entry in my grub menu.

Unfortunately, by default update-grub does not allow to customize the rescue menu entry options. I have just filed #766530 hoping for that to change.

After testing the patch I proposed for /etc/grub.d/10_linux, I now have this in my /etc/default/grub, with some satisfaction:

GRUB_CMDLINE_LINUX_RECOVERY="systemd.log_target=kmsg systemd.log_level=debug systemd.debug-shell"

Further information:

Thanks to sjoerd and uau on #debian-systemd for their help.

Posted Thu Oct 23 22:06:30 2014 Tags:

Spelling a chilometri zero

Lo spelling internazionale è troppo globalizzato, e volete recuperare un attimo la dimensione del posto dove siete nati e cresciuti?

Da oggi c'è questo script che fa per voi: gli dite dove abitate, e lui vi crea lo spelling a chilometri zero.

$ git clone git@gitorious.org:trespolo/osmspell.git
$ cd osmspell
$ ./osmspell "San Giorgio di Piano"
1: San Giorgio di Piano, BO, EMR, Italia
2: San Giorgio di Piano, Via Codronchi, San Giorgio di Piano, BO, EMR, Italia
3: San Giorgio Di Piano, Via Libertà, San Giorgio di Piano, BO, EMR, Italia
Choose one: 1
Center: 44.6465332, 11.3790398
A Argelato, Altedo
B Bentivoglio, Bologna, Boschi
C Cinquanta, Castagnolo Minore, Castel Maggiore, Cento
D Dosso
E Eremo di Tizzano
F Funo di Argelato, Finale Emilia, Ferrara, Fiesso
G Gherghenzano, Galliera, Gesso
I Il Cucco, Irnerio, Idice
L Località Fortuna, Lovoleto, Lippo
M Malacappa, Massumatico, Minerbio, Marano
N Navile
O Osteriola, Ozzano dell'Emilia, Oca
P Piombino, Padulle, Poggio Renatico, Piave
Q Quarto Inferiore, Quattrina
R Rubizzano, Renazzo, Riale
S San Giorgio di Piano, Saletto
T Torre Verde, Tintoria, Tombe
U Uccellino
V Venezzano Mascarino, Vigarano Mainarda, Veduro
X XII Morelli
Z Zenerigolo, Zola Predosa

I dati vengono da OSM, e lo script è un ottimo esempio di come usarne la API di geolocazione (veloci) e la API di query geografica (lenta).

Posted Sat Jan 4 00:38:16 2014 Tags:
Posted Tue Apr 7 19:20:12 2015

Cazzeggio.

Non importa che mi dai del voi

Dai, non importa che mi dai del voi

In che senso?

Eh, mi dici sempre "voi informatici", "voi tecnici", "voi..."

Posted Fri Dec 19 15:55:20 2014 Tags:

Spelling a chilometri zero

Lo spelling internazionale è troppo globalizzato, e volete recuperare un attimo la dimensione del posto dove siete nati e cresciuti?

Da oggi c'è questo script che fa per voi: gli dite dove abitate, e lui vi crea lo spelling a chilometri zero.

$ git clone git@gitorious.org:trespolo/osmspell.git
$ cd osmspell
$ ./osmspell "San Giorgio di Piano"
1: San Giorgio di Piano, BO, EMR, Italia
2: San Giorgio di Piano, Via Codronchi, San Giorgio di Piano, BO, EMR, Italia
3: San Giorgio Di Piano, Via Libertà, San Giorgio di Piano, BO, EMR, Italia
Choose one: 1
Center: 44.6465332, 11.3790398
A Argelato, Altedo
B Bentivoglio, Bologna, Boschi
C Cinquanta, Castagnolo Minore, Castel Maggiore, Cento
D Dosso
E Eremo di Tizzano
F Funo di Argelato, Finale Emilia, Ferrara, Fiesso
G Gherghenzano, Galliera, Gesso
I Il Cucco, Irnerio, Idice
L Località Fortuna, Lovoleto, Lippo
M Malacappa, Massumatico, Minerbio, Marano
N Navile
O Osteriola, Ozzano dell'Emilia, Oca
P Piombino, Padulle, Poggio Renatico, Piave
Q Quarto Inferiore, Quattrina
R Rubizzano, Renazzo, Riale
S San Giorgio di Piano, Saletto
T Torre Verde, Tintoria, Tombe
U Uccellino
V Venezzano Mascarino, Vigarano Mainarda, Veduro
X XII Morelli
Z Zenerigolo, Zola Predosa

I dati vengono da OSM, e lo script è un ottimo esempio di come usarne la API di geolocazione (veloci) e la API di query geografica (lenta).

Posted Sat Jan 4 00:38:16 2014 Tags:

Poesia: "Lavatrice"

Pensavo fosse pail,

invece ora è feltro.

Posted Tue Dec 3 22:32:23 2013 Tags:

Shops

Christmas songs should only ever be played on Christmas day.

In church.

At midnight.

Unless I happen to be there.

Posted Mon Dec 2 14:07:58 2013 Tags:

Airports

Photo of a commercial in London City airport saying 'In the lap of luxury - Want to reach a captive audience with dwell time? Why advertise anywhere else? - London City Airport Media Sales'

In the airport, we are not travellers. We are a captive audience with dwell time.

In other words, suckers stuck in a room where the only pastime provided is spending money and staring at advertisements selling advertisement space in rooms full of suckers like them.

Posted Fri Nov 22 18:58:00 2013 Tags:

Explanation of umarell

Umarell /uma'rɛl/ (oo-mah-rell), n; pl. Umarells. People in a community who offer all sorts of comments to those who are trying to get some work done, but who are not doing any work themselves.

Etymology and further details

Umarell is a word that entered Italian slang in Bologna and is spreading to nearby towns, occasionally even across Italy. It comes from the Bolognese for "cute/odd little man".

"Umarells" are those people, usually retired men, who spend time watching construction works, often holding their hands behind their back, occasionally commenting on what is going on, sometimes trying to tell the workers what to do.

It's easy to find examples on the internet; the word was popularised by a blog collecting photos, which has even been published into a book.

With some Italian Debian friends, we realised that umarell is the perfect word to describe those people in a community, who offer all sorts of comments to those who are trying to get some work done, but who are not doing any work themselves.

I think that it is a word that fits perfectly, and since I'm likely going to use it blissfully anywhere, here is a page that temporarily explains what it means until the Oxford English Dictionary picks it up.

Posted Fri Sep 20 13:27:07 2013 Tags:

Yet another Ubuntu anecdote

Some posts on planet made me remember of a little Canonical-related story of mine.

Many years ago I shortly contracted for Canonical. It was interesting and fun.

At the time I didn't have any experience of being temporarily hired by a foreign company, so I rang my labour union to get an appointment, to make sure with them that everything was allright.

The phone call went more or less like this:

Me:

Hello. I have received this contract for temporary employment by a foreign company and I wondered if I could book an appointment to come show it to you to see if it's all ok.

Their answer rather cut me short:

Hi. Be careful! People get temporary employment from obscure companies with the headquarters, like, in the Isle of Man, they do the job, the company disappears and they never get paid. There's bad stuff out there!

I looked at the contract, the heading said something like "Canonical ltd, Douglas, Isle of Man".

I was certain that the union people would have never understood what was going on. I politely thanked them for their time and hung up. However, to this day I still regret that I didn't insist:

Uh, yes, the company is indeed in the Isle of Man. But what if I told you that it's owned by an astronaut?

I just signed the contract and had a good time.

Posted Sat Jan 15 10:35:36 2011 Tags:

Mailman defaults

Monopoly Chance: It's the first of the month. / You're flooded with mailman junk / Skip a turn.

Posted Fri Oct 1 12:03:53 2010 Tags:

My rule to see if a framework is worth of attention

I came up with a little rule:

In order to be worth of any attention, a framework must be stable enough that I can charge money to train people to use it.

This probably applies to other kinds of software stacks, libraries, development environments and, well, to most software applications.

In the context of python web frameworks, this means that:

  • If it changes API all the time it is not worth of attention, because my customers won't get value for their money, as they'd continuously need retraining and rewriting their software.
  • If I see lots of DeprecationWarnings it is not worth of attention, because my customers will see them and blame me for teaching them deprecated stuff.
  • If fixes for bugs affecting the stable version are only distributed "in a recent git" or "in the next development version", and they are not backported into a new bugfix-only stable release, then it is not worth of attention, because:
    • my customers' business is to develop their own products based on the framework.
    • My customers' business is not to be maintaning in-house stable updates of the framework. Although if the framework's community is nice enough they might end up giving a hand.
  • If it requires virtualenv or can only be obtained through easy_install it is not worth of attention, because:
    • my customers are not interested in maintaning custom deployment environments over time.
    • My customers are not interested in tracking each and every single library's upstream development to keep their production system free of bugs.
    • My customers are used to getting software through a proper distribution which also takes care of security updates.
    • I am paid to teach them how to use a framework, not a custom python-only package management system.
    • In my experience, if distributions have trouble keeping packages up to date, upstream is doing something fundamentally wrong.

In light of this rule, I regret to notice that I see very few python web frameworks worth of any attention.

Posted Wed Aug 4 15:32:24 2010 Tags:

On python stable APIs

There is a theory which states that if ever anyone discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable.

There is another theory which states that this has already happened.

In Debian testing:

/usr/lib/python2.6/dist-packages/sqlalchemy/types.py:547: SADeprecationWarning: The Binary type has been renamed to LargeBinary.

In Debian Lenny:

ImportError: cannot import name LargeBinary

I was starting to think that SQLAlchemy wasn't too bad, since I've been using it for 6 months and I haven't seen its API change yet.

But there it is, a beautiful reminder that SQLAlchemy, too, is part of the marvelously autistic Python ecosystem.

Posted Mon Jul 19 16:14:25 2010 Tags:
Posted Sat Jun 6 00:57:39 2009

Rants, kept to a bare minimum and strictly with a useful component.

Another day in the life of a poor developer

try:
    # After Python 3.3
    from collections.abc import Iterable
except ImportError:
    # This has changed in Python 3.3 (why, oh why?), reinforcing the idea that
    # the best Python version ever is still 2.7, simply because upstream has
    # promised that they won't touch it (and break it) for at least 5 more
    # years.
    from collections import Iterable

import shlex
if hasattr(shlex, "quote"):
    # New in version 3.3.
    shell_quote = shlex.quote
else:
    # Available since python 1.6 but deprecated since version 2.7: Prior to Python
    # 2.7, this function was not publicly documented. It is finally exposed
    # publicly in Python 3.3 as the quote function in the shlex module.
    #
    # Except everyone was using it, because it was the only way provided by the
    # python standard library to make a string safe for shell use
    #
    # See http://stackoverflow.com/questions/35817/how-to-escape-os-system-calls-in-python
    import pipes
    shell_quote = pipes.quote

import shutil
if hasattr(shutil, "which"):
    # New in version 3.3.
    shell_which = shutil.which
else:
    # Available since python 1.6:
    # http://stackoverflow.com/questions/377017/test-if-executable-exists-in-python
    from distutils.spawn import find_executable
    shell_which = find_executable
Posted Fri Feb 27 12:02:33 2015 Tags:

Akonadi client example

After many failed attemps I have managed to build a C++ akonadi client. It has felt like one of the most frustrating programming experiences of my whole life, so I'm sharing the results hoping to spare others from all the suffering.

First thing first, akonadi client libraries are not in libakonadi-dev but in kdepimlibs5-dev, even if kdepimlibs5-dev does not show in apt-cache search akonadi.

Then, kdepimlibs is built with Qt4. If your application uses Qt5 (mine was) you need to port it back to Qt4 if you want to talk to Akonadi.

Then, kdepimlibs does not seem to support qmake and does not ship pkg-config .pc files, and if you want to use kdepimlibs your build system needs to be cmake. I ported by code from qmake to cmake, and now qtcreator wants me to run cmake by hand every time I change the CMakeLists.txt file, and it stopped allowing to add, rename or delete sources.

Finally, most of the code / build system snippets found on the internet seem flawed in a way or another, because the build toolchain of Qt/KDE applications has undergone several redesignins during time, and the network is littered with examples from different eras. The way to obtain template code to start a Qt/KDE project is to use kapptemplate. I have found no getting started tutorial on the internet that said "do not just copy the snippets from here, run kapptemplate instead so you get them up to date".

kapptemplate supports building an "Akonadi Resource" and an "Akonadi Serializer", but it does not support generating template code for an akonadi client. That left me with the feeling that I was dealing with some software that wants to be developed but does not want to be used.

Anyway, now an example of how to interrogate Akonadi exists as is on the internet. I hope that all the tears of blood that I cried this morning have not been cried in vain.

Posted Mon Feb 23 15:44:01 2015 Tags:

The wonders of missing documentation

Update: I have managed to build an example Akonadi client application.

I'm new here, I want to make a simple C++ GUI app that pops up a QCalendarWidget which my local Akonadi has appointments.

I open qtcreator, create a new app, hack away for a while, then of course I get undefined references for all Akonadi symbols, since I didn't tell the build system that I'm building with akonadi. Ok.

How do I tell the build system that I'm building with akonadi? After 20 minutes of frantic looking around the internet, I still have no idea.

There is a package called libakonadi-dev which does not seem to have anything to do with this. That page mentions everything about making applications with Akonadi except how to build them.

There is a package called kdepimlibs5-dev which looks promising: it has no .a files but it does have haders and cmake files. However, qtcreator is only integrated with qmake, and I would really like the handholding of an IDE at this stage.

I put something together naively doing just what looked right, and I managed to get an application that segfaults before main() is even called:

/*
 * Copyright © 2015 Enrico Zini <enrico@enricozini.org>
 *
 * This work is free. You can redistribute it and/or modify it under the
 * terms of the Do What The Fuck You Want To Public License, Version 2,
 * as published by Sam Hocevar. See the COPYING file for more details.
 */
#include <QDebug>

int main(int argc, char *argv[])
{
    qDebug() << "BEGIN";
    return 0;
}
QT       += core gui widgets
CONFIG += c++11

TARGET = wtf
TEMPLATE = app

LIBS += -lkdecore -lakonadi-kde

SOURCES += wtf.cpp

I didn't achieve what I wanted, but I feel like I achieved something magical and beautiful after all.

I shall now perform some haruspicy on those oscure cmake files to see if I can figure something out. But seriously, people?

Posted Mon Feb 23 11:36:18 2015 Tags:

Mozilla marketplace facepalm

This made me sad.

My view, which didn't seem to be considered in that discussion, is that people concerned about software freedom and security are likely to stay the hell away from such an app market and its feedback forms.

Also, that thread made me so sad about the state of that developer community that I seriously do not feel like investing energy into going through the hoops of getting an account in their bugtracker to point this out.

Sigh.

Posted Fri Jan 23 15:13:16 2015 Tags:

SSL "protection"

In my experience with my VPS, setting up pretty much any service exposed to the internet, even a simple thing to put a calendar in my phone requires an SSL certificate, which costs money, which needs to be given to some corporation or another.

When the only way to get protection from a threat is to give money to some big fish, I feel like I'm being forced to pay protection money.

I look forward to this.

Posted Thu Dec 11 15:35:25 2014 Tags:

Laptop, I demand that you suspend!

Dear Lazyweb,

Sometimes some application prevents suspend on my laptop. I want to disable that feature: how?

I understand that there may exist some people who like that feature. I, on the other hand, consider a scenario like this inconceivable:

  1. I'm on a plane working with my laptop, the captain announces preparations for landing, so I quickly hit the suspend button (or close the lid) on my laptop and stow it away.
  2. One connecting flight later, I pick up my backpack, I feel it unusually hot and realise that my laptop has been on all along, and is now dead from either running out of battery or thermal protection.
  3. I think things that, if spoken aloud in front of a pentacle, might invoke major lovecraftian horrors.

I do not want this scenario to ever be possible. I want my suspend button to suspend the laptop no matter what. If a process does not agree, I'm fine with suspending it anyway, or killing it.

If I want my laptop to suspend, I generally have a good enough real-world reason for it, and I cannot conceive that a software could ever be allowed to override my command.

How do I change this? I don't know if I should look into systemd, upowerd, pm-utils, the kernel, the display manager or something else entirely. I worry that I cannot even figure where to start looking for a solution.

This happened to me multiple times already, and I consider it ridiculous. I know that it can cause me data loss. I know that it can cause me serious trouble in case I was relying on having some battery or state left at my arrival. I know that depending on what is in my backpack, this could also be physically dangerous.

So, what knob do I tweak for this? How do I make suspend reliable?

Update

Systemd has an inhibitor system, and systemd-inhibit --list only lists 'delay' blocks in my system. It is an interesting feature that seems to be implemented in the right way, and it could mean that I finally can get my screen to be locked before the system is suspended.

It is possible to configure the inhibitor system in /etc/systemd/logind.conf, including ways to ignore inhibitors, and a maximum time after which inhibitors are ignored if not yet released.

Try as I might to run everything that I was running on the plane that time, I could not manage to see anything take an inhibitor block that could have prevented my suspend. I now suspect that what happened to me was a glitch caused by something else (hardware? kernel? cosmic rays!) during that specific suspend.

When I had this issue in the past it looks like the infrastructure at the time was far more primitive that what we have now with systemd, so I guess that when writing my blog post I had simply correlated my old experiences with a one-off suspend glitch.

If I want to investigate or tune further, to test the situation with a runaway block, I can use commands like systemd-inhibit --mode=block sleep 3600.

I'm quite happy to see that we're moving to a standard and sane system for this. In the meantime, I have learnt that pm-utils has now become superfluous and can be deinstalled, and so can acpi-support and acpi-support-base.

Thanks vbernat, mbiebl, and ah, on #debian-devel for all the help.

Posted Thu Sep 11 14:32:40 2014 Tags:

Wheezy for industrial software development

I'm helping with setting up a wheezy-based toolchain for industrial automation.

The basic requirements are: live-build, C++11, Qt 5.3, and a frozen internal wheezy mirror.

debmirror

A good part of a day's work was lost because of #749734 and possibly #628779. Mirror rebuild is still ongoing, and fingers crossed.

This is Italy, and you can't simply download 21Gb of debs just to see how it goes.

C++11

Stable toolchains for C++11 now exist and have gained fast adoption. It makes sense, since given what is in C++11 it is unthinkable to start a new C++ project with the old standard nowadays.

C++11 is supported by g++ 4.8+ or clang 3.3+. None of them is available on wheezy or wheezy-backports.

Backports exist of g++ 4.8 only for Ubuntu 12.04, but they are uninstallable on wheezy due at least to a different libc6. I tried rebuilding g++4.8 on wheezy but quickly gave up.

clang 3.3 has a build dependency on g++ 4.8. LOL.

However, LLVM provides an APT repository with their most recent compiler, and it works, too. C++11 problem solved!

Qt 5.3

Qt 5.3 is needed because of the range of platforms it can target. There is no wheezy backport that I can find.

I cannot simply get it from Qt's Download page and install it, since we need it packaged, to build live ISOs with it.

I'm attempting to backport the packages from experimental to wheezy.

Here are its build dependencies:

libxcb-1.10 (needed by qt5)

Building this is reasonably straightforward.

libxkbcommon 0.4.0 (needed by qt5)

The version from jessie builds fine on wheezy, provided you remove --fail-missing from the dh_install invocation.

libicu 52.1 (needed by harfbuzz)

The jessie packages build on wheezy, provided that mentions of clang are deleted from source/configure.ac, since it fails to build with clang 3.5 (the one currently available for wheezy on llvm.org).

libharfbuzz-dev

Backporting this is a bloodbath: the Debian packages from jessie depend on a forest of gobject hipsterisms of doom, all unavailable on wheezy. I gave up.

qt 5.3

qtbase-opensource-src-5.3.0+dfsg can be made to build with an embedded version of harfbuzz, with just this change:

diff -Naur a/debian/control a/debian/control
--- a/debian/control    2014-05-20 18:48:27.000000000 +0200
+++ b/debian/control    2014-05-29 17:45:31.037215786 +0200
@@ -28,7 +28,6 @@
                libgstreamer-plugins-base0.10-dev,
                libgstreamer0.10-dev,
                libgtk2.0-dev,
-               libharfbuzz-dev,
                libicu-dev,
                libjpeg-dev,
                libmysqlclient-dev,
diff -Naur a/debian/rules b/debian/rules
--- a/debian/rules  2014-05-18 01:56:37.000000000 +0200
+++ b/debian/rules  2014-05-29 17:45:25.738634371 +0200
@@ -108,7 +108,6 @@
                -plugin-sql-tds \
                -system-sqlite \
                -platform $(platform_arg) \
-               -system-harfbuzz \
                -system-zlib \
                -system-libpng \
                -system-libjpeg \

(thanks Lisandro Damián Nicanor Pérez Meyer for helping me there!)

There are probably going to be further steps in the Qt5 toolchain.

Actually, let's try prebuilt binaries

The next day with a fresh mind we realised that it is preferable to reduce our tampering with the original wheezy to a minimum. Our current plan is to use wheezy's original Qt and Qt-using packages, and use Qt's prebuilt binaries in /opt for all our custom software.

We run Qt's installer, tarred the result, and wrapped it in a Debian package like this:

$ cat debian/rules
#!/usr/bin/make -f

QT_VERSION = 5.3

%:
    dh $@

override_dh_auto_build:
    dh_auto_build
    sed -re 's/@QT_VERSION@/$(QT_VERSION)/g' debian-rules.inc.in > debian-rules.inc

override_dh_auto_install:
    dh_auto_install
    # Download and untar the prebuild Qt5 binaries
    install -d -o root -g root -m 0755 debian/our-qt5-sdk/opt/Qt
    curl http://localserver/Qt$(QT_VERSION).tar.xz | xz -d | tar -C debian/our-qt5-sdk/opt -xf -
    # Move the runtime part to our-qt5
    install -d -o root -g root -m 0755 debian/our-qt5/opt/Qt
    mv debian/our-qt5-sdk/opt/Qt/$(QT_VERSION) debian/our-qt5/opt/Qt/
    # Makes dpkg-shlibdeps work on packages built with Qt from /opt
    # Hack. Don't try this at home. Don't ever do this unless you
    # know what you are doing. This voids your warranty. If you
    # know what you are doing, you won't do this.
    find debian/our-qt5/opt/Qt/$(QT_VERSION)/gcc_64/lib -maxdepth 1 -type f -name "lib*.so*" \
        | sed -re 's,^.+/(lib[^.]+)\.so.+$$,\1 5 our-qt5 (>= $(QT_VERSION)),' > debian/our-qt5.shlibs


$ cat debian-rules.inc.in
export PATH := /opt/Qt/@QT_VERSION@/gcc_64/bin:$(PATH)
export QMAKESPEC=/opt/Qt/@QT_VERSION@/gcc_64/mkspecs/linux-clang/

To build one of our packages using Qt5.3 and clang, we just add this to its debian/rules:

include /usr/share/our-qt5/debian-rules.inc

Wrap up

We got the dependencies sorted. Hopefully the mirror will rebuild itself tonight and tomorrow we can resume working on our custom live system.

Posted Thu May 29 18:05:17 2014 Tags:

Shops

Christmas songs should only ever be played on Christmas day.

In church.

At midnight.

Unless I happen to be there.

Posted Mon Dec 2 14:07:58 2013 Tags:

Airports

Photo of a commercial in London City airport saying 'In the lap of luxury - Want to reach a captive audience with dwell time? Why advertise anywhere else? - London City Airport Media Sales'

In the airport, we are not travellers. We are a captive audience with dwell time.

In other words, suckers stuck in a room where the only pastime provided is spending money and staring at advertisements selling advertisement space in rooms full of suckers like them.

Posted Fri Nov 22 18:58:00 2013 Tags:

A vision wanted

Today Richard Stallman mailed all Italian LUGs asking that tomorrow's LinuxDay be called "GNU/Linux Day" instead.

I wonder how that is ever going to help a community so balkanised, that the only way Italian LUGs manage to do something together, is to say "let's not care what we all do, let's just do it on the same day and call it a national event". Of course a few LUGs still make a point of not doing anything on that day, because you know, Judean People's Front. Cawk.

Today a friend asked me if I could help her support people in installing Citrix Whatsit to set up a video conference to network meetings that will take place in a month in different cities. Those meetings are something I look forwad to. It wasn't much of a problem to say "no, I can't do that"; it was a problem to be unable to come up with some viable, Free alternatives.

I sometimes have to use Skype to talk with friends who also are Debian Developers, because I still haven't managed to make VoIP work unless I go through a commercial proxy.

There was the happy news that our regional administration is switching from MS Word to OpenOffice. It soon became a flamewar, because some people loudly complained that they should have used LibreOffice instead.

At DebConf, after spending an hour getting frustrated with the default formatting of bullet points in WhateverOffice Impress, I did my Debian Contributors talk using a text file in vim. And it was a success! Thanks Francois Marier for maintaining cowsay.

I can't sync contact lists and appointments between my N900, which runs a Debian derivative, and my laptop, because I don't want to have a Google account, and nothing else would work out of the box.

I don't even know how to keep a shared calendar with other DDs, without using a 3rd party cloud service that I don't want to trust with my life's personal schedule.

I need to do a code review of every vim plugin I need to use, because you can only get them by cloning GitHub repositories over plain http, and they don't even provide signed tags. That code is run with my own privileges every time I start a text editor, which is, like, all the time. I'm frightened at the idea of how many people blissfully don't think about what that means. Vim users. Programmers. Cool chaps.

Yet the most important thing in Debian today seems to be yet another crusade between upstart and systemd.

But we haven't had a lengthy discussion on why, although the excellent OpenStreetMap exists and many of us contribute to it, it seems to still be more immediate to hit Google Maps to get a route computed. How can we change that?

We haven't had a lengthy discussion on what can we offer to allow anyone to set up some social platform that won't get swamped with spam the first day and cracked open the second; that would allow people to share some photos with their friends only, and some with the rest of the world; that won't required a full-time paid person to maintain. That won't be obsolete and require a migration to a new platform in a year. That isn't Facebook or Google Plus.

I stopped taking photos because it's too much work to show them to people. Other people use Instagram. Whatever the hipster trend is for photo sharing today, October 25, 2013, I'm pretty sure it's not a Free platform.

But we can do something. We technology leaders. We are those who drive technological change!

For example, today I invested two hours of hard effort trying to figure out why libbuffy's test suite fails on kfreebsd. All while wondering why I was doing that, since I know all buffy's users personally, and none of them uses kfreebsd.

And I will take a day off work to study the library symbols file specification, so that next time I'll know right away if the new version of a C++ compiler decides that a template-generated symbol isn't worth adding to a library anymore.

What is this effort really about? It sometimes feel like micromanaging to me.

It's good to have excellent quality standards. But not without a vision.

Not until "reliable network printing with all PDF viewers and print servers we ship" is among our release goals.

Not until we commit to making sure that "sharing files between Debian users" will work out of the box, without the need of going through a 3rd party website, or email.

I'm not interested in spending energy discussing init systems. I'm interested in spending energy sharing stories of what cool stuff we can do in Debian today, out of the box. And what cool stuff we'll be able to do tomorrow.

Let's spend time on IRC, on mailing lists, and at the next Debian events, talking about why we are really into this. Talking about a vision!

Note: Please don't spend time telling me how to fix the problems I mentioned above. I'm not interested in help fixing some problems for me today. I'm interested in asking help fixing problems for everybody, right in the next stable release.

Remember, remember, the 5th of November, 2014.

Posted Fri Oct 25 22:38:11 2013 Tags:
Posted Sat Jun 6 00:57:39 2009

Pages about OpenMoko.

Released nodm 0.7

I have released version 0.7 of nodm.

It only fixes one silly typo in autotools, which made it fail to build on Fedora.

Posted Sun May 23 21:36:52 2010 Tags:

Released nodm 0.6

I have released version 0.6 of nodm.

It is purely a bug fix release, trying harder to detect a console in order to get rid of a bug introduced with version 0.5

Posted Mon Aug 3 12:34:16 2009 Tags:

Released nodm 0.5

I have released version 0.5 of nodm.

New features:

  • truncate ~/.xsession-errors on startup: finally that file stops growing, and growing, and growing...
  • dynamic VT allocation: it can now avoid opening a virtual terminal if it is already in use.
Posted Fri Jul 24 02:29:55 2009 Tags:

Getting dbus signatures right from Vala

I am trying to play a bit with Vala on the FreeRunner.

The freesmartphone.org stack on the OpenMoko is heavily based on DBus. Using DBus from Vala is rather simple, if mostly undocumented: you get a few examples in the Vala wiki and you make do with those.

All works fine with simple methods. But what with providing callbacks to signals that have complex nested structures in their signatures, like aa{sv}? You try, and then if you don't get the method signature right, the signal is just silently not delivered because it does not match the method signature.

So this is how to provide a callback to org.freesmartphone.Usage.ResourceChanged, with signature sba{sv}:

public void on_resourcechanged(dynamic DBus.Object pos,
                   string name,
                   bool state,
                   HashTable<string, Value?> attributes)
{
    stderr.printf("Resource %s changed\n", name);
}

And this is how to provide a callback to org.freesmartphone.GPS.UBX.DebugPacket, with signature siaa{sv}:

protected void on_ubxdebug_packet(dynamic DBus.Object ubx, string clid, int length,
        HashTable<string, Value?>[] wrongdata)
{
    stderr.printf("Received UBX debug packet");

    // Ugly ugly work-around
    PtrArray< HashTable<string, Value?> >* data = (PtrArray< HashTable<string, Value?> >)wrongdata;

    stderr.printf("%u elements received", data->len);
}

What is happening here is that the only method signature that I found matching the dbus signature is this one. However, the unmarshaller for some reason gets it wrong, and passes a PtrArray instead of a HashTable array. So you need to cast it back to what you've actually been passed.

Figuring all this out took several long hours and was definitely not fun.

Posted Wed Jul 15 12:30:50 2009 Tags:

Mapping using the Openmoko FreeRunner headset

The FreeRunner has a headset which includes a microphone and a button. When doing OpenStreetMap mapping, it would be very useful to be able to keep tangogps on the display and be able to mark waypoints using the headset button, and to record an audio track using the headset microphone.

In this way, I can use tangogps to see where I need to go, where it's already mapped and where it isn't, and then I can use the headset to mark waypoints corresponding to the audio track, so that later I can take advantage of JOSM's audio mapping features.

Enter audiomap:

$ audiomap --help
Usage: audiomap [options]

Create a GPX and audio trackFind the times in the wav file when there is clear
voice among the noise

Options:
  --version      show program's version number and exit
  -h, --help     show this help message and exit
  -v, --verbose  verbose mode
  -m, --monitor  only keep the GPS on and monitor satellite status
  -l, --levels   only show input levels

If called without parameters, or with -v which is suggested, it will:

  1. Fix the mixer settings so that it can record from the headset and detect headset button presses.
  2. Show a monitor of GPS satellite information until it gets a fix.
  3. Synchronize the system time with the GPS time so that the timestamps of the files that are created afterwards are accurate.
  4. Start recording a GPX track.
  5. Start recording audio.
  6. Record a GPX waypoint for every headset button press.

When you are done, you stop audiomap with ^C and it will properly close the .wav file, close the tags in the GPX waypoint and track files and restore the mixer settings.

You can plug the headset out and record using the handset microphone, but then you will not be able to set waypoints until you plug the headset back in.

After you stop audiomap, you will have a track, waypoints and .wav file ready to be loaded in JOSM.

Big thanks go to Luca Capello for finding out how to detect headset button presses.

Posted Sun Jun 7 23:51:37 2009 Tags:

Simple tool to query the GPS using the OpenMoko FSO stack

I was missing a simple command line tool that allows me to perform basic GPS queries in shellscripts.

Enter getgps:

# getgps --help
Usage: getgps [options]

Simple GPS query tool for the FSO stack

Options:
  --version          show program's version number and exit
  -h, --help         show this help message and exit
  -v, --verbose      verbose mode
  -q, --quiet        suppress normal output
  --fix              check if we have a fix
  -s, --sync-time    set system time from GPS time
  --info             get all GPS information
  --info-connection  get GPS connection information
  --info-fix         get GPS fix information
  --info-position    get GPS position information
  --info-accuracy    get GPS accuracy information
  --info-course      get GPS course information
  --info-time        get GPS time information
  --info-satellite   get GPS satellite information

So finally I can write little GPS-aware scripts:

if getgps --fix -q
then
    start_gps_aware_program
else
    start_gps_normal_program
fi

Or this.

Posted Sun Jun 7 17:59:32 2009 Tags:

Voice-controlled waypoints

I have it in my TODO list to implement taking waypoints when pressing the headset button of the openmoko, but that is not done yet.

In the meantime, I did some experiments with audio mapping, and since I did not manage to enter waypoints while recording them, I was looking for a way to make use of them anyway.

Enter findvoice:

$ ./findvoice  --help
Usage: findvoice [options] wavfile

Find the times in the wav file when there is clear voice among the noise

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -v, --verbose         verbose mode
  -p NUM, --percentile=NUM
            percentile to use to discriminate noise from voice
            (default: 90)
  -t, --timestamps      print timestamps instead of human readable information

You give it a wav file, and it will output a list of timestamps corresponding to where it things that you were talking clearly and near the FreeRunner / voice recorder instead of leaving the recorder dangling to pick up background noise.

Its algorithm is crude and improvised because I have no background whatsoever in audio processing, but it basically finds those parts of the audio file where the variance of the samples is above a given percentile: the higher the percentile, the less timestamps you get; the lower the percentile, the more likely it is that it picks a period of louder noise.

For example, you can automatically extract waypoints out of an audio file by using it together with gpxinterpolate:

./findvoice -t today.wav | ./gpxinterpolate today.gpx > today-waypoints.gpx

The timestamps it outputs are computed using the modification time of the .wav file: if your system clock was decently synchronised (which you can do with getgps), then the mtime of the wav is the time of the end of the recording, which gives the needed reference to compute timestamps that are absolute in time.

For example:

getgps --sync-time
arecord file.wav
^C
./findvoice -t file.wav | ./gpxinterpolate today.gpx > today-waypoints.gpx
Posted Sun Jun 7 02:48:40 2009 Tags:

Geocoding Unix timestamps

Geocoding EXIF tags in JPEG images is fun, but there is more that can benefit from interpolating timestamps over a GPX track.

Enter gpxinterpolate:

$ ./gpxinterpolate --help
Usage: gpxinterpolate [options] gpxfile [gpxfile...]

Read one or more GPX files and a list of timestamps on standard input. Output
a GPX file with waypoints at the location of the GPX track at the given
timestamps.

Options:
  --version      show program's version number and exit
  -h, --help     show this help message and exit
  -v, --verbose  verbose mode

For example, you can create waypoints interpolating file modification times:

find . -printf "%Ts %p\n" | ./gpxinterpolate ~/tracks/*.gpx > myfiles.gpx

In case you wonder where you were when you modified or accessed a file, now you can find out.

Posted Sun Jun 7 02:07:43 2009 Tags:

Recording audio on the FreeRunner

The FreeRunner can record audio. It is nice to record audio: for example I can run the recording in background while I keep tangogps in the screen, and take audio notes about where I am while I am doing mapping for OpenStreetMap.

Here is the script that I put together to create geocoded audio notes:

#!/bin/sh

WORKDIR=~/rec
TMPINFO=`mktemp $WORKDIR/info.XXXXXXXX`

# Sync system time and get GPS info
echo "Synchronising system time..."
getgps --sync-time --info > $TMPINFO

# Compute an accurate basename for the files we generate
BASENAME=~/rec/rec-$(date +%Y-%m-%d-%H-%M-%S)
# Then give a proper name to the file with saved info
mv $TMPINFO $BASENAME.info

# Proper mixer settings for recording
echo "Recording..."
alsactl -f /usr/share/openmoko/scenarios/voip-handset.state restore
arecord -D hw -f cd -r 8000 -t wav $BASENAME.wav

echo "Done"

It works like this:

  1. It synchronizes the system time from the GPS (if there is a fix) so that the timestamps on the wav files will be as accurate as possible.
  2. It also gets all sort of information from the GPS and stores them into a file, should you want to inspect it later.
  3. It records audio until it gets interrupted.

The file name of the files that it generates corresponds to the beginning of the recording. The mtime of the wav file obviously corresponds to the end of the recording. This can be used to later georeference the start and end point of the recording.

You can use this to check mixer levels and that you're actually getting any input:

arecord -D hw -f cd -r 8000 -t wav -V mono /dev/null

The getgps script is now described in its own post.

You may now want to experiment, in JOSM, with "Preferences / Audio settings / Modified times (time stamps) of audio files".

Posted Sun Jun 7 01:30:37 2009 Tags:

How to read the Freerunner's accelerometers

This code has been take from moko_eightball by Jakob Westhoff: it just continuously prints the value of the three accelerometers.

#include <stdio.h>
#include <stdint.h>

void processInputEvents(FILE* in)
{
    int x = 0, y = 0, z = 0;
    while (1)
    {
        char padding[16];
        uint16_t type, code;
        int32_t value;

        // Skip the timestamp
        fread(padding, 1, 8, in);

        // Read the type
        fread(&type, 1, 2, in);

        // Read the code
        fread(&code, 1, 2, in);

        // Read the value
        fread(&value, 1, 4, in);

        switch( type )
        {
            case 0:
                switch( code )
                {
                    case 0:
                        fprintf(stdout, "x%d y%d z%d\n", x, y, z);
                        break;
                    default:
                        //warning( "Unknown code ( 0x%02x ) for type 0x%02x\n", code, type );
                        break;
                }
                break;
            case 2:
                switch ( code )
                {
                    case 0:
                        // Update to the new value
                        x = value;
                        break;
                    case 1:
                        // Update to the new value
                        y = value;
                        break;
                    case 2:
                        // Update to the new value
                        z = value;
                        break;
                    default:
                        //warning( "Unknown code ( 0x%02x ) for type 0x%02x\n", code, type );
                        break;
                }
                break;

            default:
                //warning( "Unknown type ( 0x%02x ) in accelerometer input stream\n", type );
                break;
        }


    }
}


int main()
{
    FILE* in = fopen("/dev/input/event2", "r");
    processInputEvents(in);
    fclose(in);
    return 0;
}
Posted Sat Jun 6 00:57:39 2009 Tags:
Posted Sat Jun 6 00:57:39 2009
sw

Software

Work around Google evil .ics feeds

I've happily been using 2015/akonadi-install for my calendars, and yesterday I added an .ics feed export from Google, as a URL file source. It is a link in the form: https://www.google.com/calendar/ical/person%40gmail.com/private-12341234123412341234123412341234/basic.ics

After doing that, I noticed that the fan in my laptop was on more often than usual, and I noticed that akonadi-server and postgres were running very often, and doing quite a lot of processing.

The evil

I investigated and realised that Google seems to be doing everything they can to make their ical feeds hard to sync against efficiently. This is the list of what I have observed Gmail doing to an unchanged ical feed:

  • Date: headers in HTTP replies are always now
  • If-Modified-Since: is not supported
  • DTSTAMP of each element is always now
  • VTIMEZONE entries appear in random order
  • ORGANIZER CN entries randomly change between full name and plus.google.com user ID
  • ATTENDEE entries randomly change between having a CN or not having it
  • TRIGGER entries change spontaneously
  • CREATED entries change spontaneously

This causes akonadi to download and reprocess the entire ical feed at every single poll, and I can't blame akonadi for doing it. In fact, Google is saying that there is a feed with several years worth of daily appointments that all keep being changed all the time.

The work-around

As a work-around, I have configured the akonadi source to point at a local file on disk, and I have written a script to update the file only if the .ics feed has actually changed.

Have a look at the script: I consider it far from trivial, since it needs to do a partial parsing of the .ics feed to throw away all the nondeterminism that Google pollutes it with.

The setup

The script needs to be run periodically, and I used it as an opportunity to try systemd user timers:

    $ cat ~/.config/systemd/user/update-ical-feeds.timer
    [Unit]
    Description=Updates ical feeds every hour
    # Only run when on AC power
    ConditionACPower=yes

    [Timer]
    # Run every hour
    OnActiveSec=1h
    # Run a minute after boot
    OnBootSec=1m
    Unit=update-ical-feeds.service

    $ cat ~/.config/systemd/user/update-ical-feeds.service
    [Unit]
    Description=Update ICal feeds

    [Service]
    # Use oneshot to prevent two updates being run in case the previous one
    # runs for more time than the timer interval
    Type=oneshot
    ExecStart=/home/enrico/tmp/calendars/update

    $ systemctl --user start update-ical-feeds.timer
    $ systemctl --user list-timers
    NEXT                         LEFT       LAST                         PASSED UNIT                    ACTIVATES
    Wed 2015-03-25 22:19:54 CET  59min left Wed 2015-03-25 21:19:54 CET  2s ago update-ical-feeds.timer update-ical-feeds.service

    1 timers listed.
    Pass --all to see loaded but inactive timers, too.

To reload the configuration after editing: systemctl --user daemon-reload.

Further investigation

I wonder if ConditionACPower needs to be in the .timer or in the .service, since there is a [Unit] section is in both. Update: I have been told it can be in the .timer.

I also wonder if there is a way to have the timer trigger only when online. There is a network-online.target and I do not know if it is applicable. I also do not know how to ask systemd if all the preconditions are currently met for a .service/.timer to run.

Finally, I especially wonder if it is worth hoping that Google will ever make their .ics feeds play nicely with calendar clients.

Posted Wed Mar 25 21:50:21 2015 Tags:

Screen-dependent window geometry

I have an external monitor for my laptop in my work desk at home, and when I work I keep a few windows like IRC on my laptop screen, and everything else on the external monitor. Then maybe I transfer on the sofa to watch a movie or in the kitchen to cook, and I unplug from the external monitor to bring the laptop with me. Then maybe I go back to the external monitor to resume working.

The result of this (with openbox) is that when I disconnect the external monitor all the windows on my external monitor get moved to the right edge of the laptop monitor, and when I reconnect the external monitor I need to rearrange them all again.

I would like to implement something that does the following:

  1. it keeps a dictionary mapping screen geometry to window geometries
  2. every time a window geometry and virtual desktop number changes, it gets recorded in the hash for the current screen geometry
  3. every time the screen geometry changes, for each window, if there was a saved window geometry + wirtual desktop number for it for the new screen geometry, it gets restored.

Questions:

  1. Is anything like this already implemented? Where?
  2. If not, what would be a convenient way to implement it myself, ideally in a wmctrl-like way that does not depend on a specific WM?

Note: I am not interested in switching to a different WM unless it is openbox with this feature implemented in it.

Posted Mon Mar 16 21:29:36 2015 Tags:

Reuse passwords in /etc/crypttab

Today's scenario was a laptop with an SSD and a spinning disk, and the goal was to deploy a Debian system on it so that as many things as possible are encrypted.

My preferred option for it is to setup one big LUKS partition in each disk, and put a LVM2 Physical Volume inside each partition. At boot, the two LUKS partition are opened, their contents are assembled into a Volume Group, and I can have everything I want inside.

This has advantages:

  • if any of the disks breaks, the other can still be unlocked, and it should still be possible to access the LVs inside it
  • once boot has happened, any layout of LVs can be used with no further worries about encryption
  • I can use pvmove to move partitions at will between SSD and spinning disks, which means I can at anytime renegotiate the tradeoffs between speed and disk space.

However, by default this causes cryptsetup to ask for the password once for each LUKS partition, even if the passwords are the same.

Searching for ways to mitigate this gave me unsatisfactory results, like:

  • decrypt the first disk, and use a file inside it as the keyfile to decrypt the second one. But in this case if the first disk breaks, I also lose the data in the second disk.
  • reuse the LUKS session key for the first disk in the second one. Same problem as before.
  • put a detached LUKS header in /boot and use it for both disks, then make regular backups of /boot. It is an interesting option that I have not tried.

The solution that I found was something that did not show up in any of my search results, so I'm documenting it here:

    # <target name> <source device>   <key file>   <options>
    ssd             /dev/sda2         main         luks,initramfs,discard,keyscript=decrypt_keyctl
    spin            /dev/sdb1         main         luks,initramfs,keyscript=decrypt_keyctl

This caches each password for 60 seconds, so that it can be reused to unlock other devices that use it. The documentation can be found at the beginning of /lib/cryptsetup/scripts/decrypt_keyctl, beware of the leopard™.

main is an arbitrary tag used to specify which devices use the same password.

This is also useful to work easily with multiple LUKS-on-LV setups:

    # <target name> <source device>          <key file>  <options>
    home            /dev/mapper/myvg-chome   main        luks,discard,keyscript=decrypt_keyctl
    backup          /dev/mapper/myvg-cbackup main        luks,discard,keyscript=decrypt_keyctl
    swap            /dev/mapper/myvg-cswap   main        swap,discard,keyscript=decrypt_keyctl
Posted Thu Mar 12 22:45:57 2015 Tags:

Another day in the life of a poor developer

try:
    # After Python 3.3
    from collections.abc import Iterable
except ImportError:
    # This has changed in Python 3.3 (why, oh why?), reinforcing the idea that
    # the best Python version ever is still 2.7, simply because upstream has
    # promised that they won't touch it (and break it) for at least 5 more
    # years.
    from collections import Iterable

import shlex
if hasattr(shlex, "quote"):
    # New in version 3.3.
    shell_quote = shlex.quote
else:
    # Available since python 1.6 but deprecated since version 2.7: Prior to Python
    # 2.7, this function was not publicly documented. It is finally exposed
    # publicly in Python 3.3 as the quote function in the shlex module.
    #
    # Except everyone was using it, because it was the only way provided by the
    # python standard library to make a string safe for shell use
    #
    # See http://stackoverflow.com/questions/35817/how-to-escape-os-system-calls-in-python
    import pipes
    shell_quote = pipes.quote

import shutil
if hasattr(shutil, "which"):
    # New in version 3.3.
    shell_which = shutil.which
else:
    # Available since python 1.6:
    # http://stackoverflow.com/questions/377017/test-if-executable-exists-in-python
    from distutils.spawn import find_executable
    shell_which = find_executable
Posted Fri Feb 27 12:02:33 2015 Tags:

Akonadi client example

After many failed attemps I have managed to build a C++ akonadi client. It has felt like one of the most frustrating programming experiences of my whole life, so I'm sharing the results hoping to spare others from all the suffering.

First thing first, akonadi client libraries are not in libakonadi-dev but in kdepimlibs5-dev, even if kdepimlibs5-dev does not show in apt-cache search akonadi.

Then, kdepimlibs is built with Qt4. If your application uses Qt5 (mine was) you need to port it back to Qt4 if you want to talk to Akonadi.

Then, kdepimlibs does not seem to support qmake and does not ship pkg-config .pc files, and if you want to use kdepimlibs your build system needs to be cmake. I ported by code from qmake to cmake, and now qtcreator wants me to run cmake by hand every time I change the CMakeLists.txt file, and it stopped allowing to add, rename or delete sources.

Finally, most of the code / build system snippets found on the internet seem flawed in a way or another, because the build toolchain of Qt/KDE applications has undergone several redesignins during time, and the network is littered with examples from different eras. The way to obtain template code to start a Qt/KDE project is to use kapptemplate. I have found no getting started tutorial on the internet that said "do not just copy the snippets from here, run kapptemplate instead so you get them up to date".

kapptemplate supports building an "Akonadi Resource" and an "Akonadi Serializer", but it does not support generating template code for an akonadi client. That left me with the feeling that I was dealing with some software that wants to be developed but does not want to be used.

Anyway, now an example of how to interrogate Akonadi exists as is on the internet. I hope that all the tears of blood that I cried this morning have not been cried in vain.

Posted Mon Feb 23 15:44:01 2015 Tags:

The wonders of missing documentation

Update: I have managed to build an example Akonadi client application.

I'm new here, I want to make a simple C++ GUI app that pops up a QCalendarWidget which my local Akonadi has appointments.

I open qtcreator, create a new app, hack away for a while, then of course I get undefined references for all Akonadi symbols, since I didn't tell the build system that I'm building with akonadi. Ok.

How do I tell the build system that I'm building with akonadi? After 20 minutes of frantic looking around the internet, I still have no idea.

There is a package called libakonadi-dev which does not seem to have anything to do with this. That page mentions everything about making applications with Akonadi except how to build them.

There is a package called kdepimlibs5-dev which looks promising: it has no .a files but it does have haders and cmake files. However, qtcreator is only integrated with qmake, and I would really like the handholding of an IDE at this stage.

I put something together naively doing just what looked right, and I managed to get an application that segfaults before main() is even called:

/*
 * Copyright © 2015 Enrico Zini <enrico@enricozini.org>
 *
 * This work is free. You can redistribute it and/or modify it under the
 * terms of the Do What The Fuck You Want To Public License, Version 2,
 * as published by Sam Hocevar. See the COPYING file for more details.
 */
#include <QDebug>

int main(int argc, char *argv[])
{
    qDebug() << "BEGIN";
    return 0;
}
QT       += core gui widgets
CONFIG += c++11

TARGET = wtf
TEMPLATE = app

LIBS += -lkdecore -lakonadi-kde

SOURCES += wtf.cpp

I didn't achieve what I wanted, but I feel like I achieved something magical and beautiful after all.

I shall now perform some haruspicy on those oscure cmake files to see if I can figure something out. But seriously, people?

Posted Mon Feb 23 11:36:18 2015 Tags:

Setting up Akonadi

Now that I have a CalDAV server that syncs with my phone I would like to use it from my desktop.

It looks like akonadi is able to sync with CalDAV servers, so I'm giving it a try.

First thing first is to give a meaning to the arbitrary name of this thing. Wikipedia says it is the oracle goddess of justice in Ghana. That still does not hint at all at personal information servers, but seems quite nice. Ok. I gave up with software having purpose-related names ages ago.

# apt-get install akonadi-server akonadi-backend-postgresql

Akonadi wants a SQL database as a backend. By default it uses MySQL, but I had enough of MySQL ages ago.

I tried SQLite but the performance with it is terrible. Terrible as in, it takes 2 minutes between adding a calendar entry and having it show up in the calendar. I'm fascinated by how Akonadi manages to use SQLite so badly, but since I currently just want to get a job done, next in line is PostgreSQL:

# su - postgres
$ createuser enrico
$ psql postgres
postgres=# alter user enrico createdb;

Then as enrico:

$ createdb akonadi-enrico
$ cat <<EOT > ~/.config/akonadi/akonadiserverrc
[%General]
Driver=QPSQL

[QPSQL]
Name=akonadi-enrico
StartServer=false
Host=
Options=
ServerPath=
InitDbPath=

I can now use kontact to connect Akonadi to my CalDAV server and it works nicely, both with calendar and with addressbook entries.

KDE has at least two clients for Akonadi: Kontact, which is a kitchen sink application similar to Evolution, and KOrganizer, which is just the calendar and scheduling component of Kontact.

Both work decently, and KOrganizer has a pretty decent startup time. I now have a usable desktop PIM application that is synced with my phone. W00T!

Next step is to port my swift little calendar display tool to use Akonadi as a back-end.

Posted Tue Feb 17 15:34:55 2015 Tags:

seat-inspect

Four months ago I wrote this somewhere:

Seeing a DD saying "this new dbus stuff scares me" would make most debian users scared. Seeing a DD who has an idea of what is going on, and who can explain it, would be an interesting and exciting experience.

So, let's be exemplary, competent and patient. Or at least, competent. Some may like or not like the changes, but do we all understand what is going on? Will we all be able to support our friends and customers running jessie?

I confess that although I understand the need for it, I don't feel competent enough to support systemd-based machines right now.

So, are we maybe in need of help, cheat sheets, arsenals of one-liners, diagnostic tools?

Maybe a round of posts on -planet like "one debian package a day" but with new features that jessie will have, and how to understand them and take advantage of them?

That was four months ago. In the meantime, I did some work, and it got better for me.

Yesterday, however, I've seen an experienced Linux person frustrated because the shutdown function of the desktop was doing nothing whatsoever. Today I found John Goerzen's post on planet.

I felt like some more diagnostic tools were needed, so I spent the day making seat-inspect.

seat-inspect tries to make the status of the login/seat system visible, to help with understanding and troubleshooting.

The intent of running the code is to have an overview of the system status, both to see what the new facilities are about, and to figure out if there is something out of place.

The intent of reading the code is to have an idea of how to use these facilities: the code has been written to be straightforward and is annotated with relevant bits from the logind API documentation.

seat-inspect is not a finished tool, but a starting point. I put it on github hoping that people will fork it and add their own extra sanity checks and warnings, so that it can grow into a standard thing to run if a system acts weird.

As it is now, it should be able to issue warnings if some bits are missing for network-manager or shutdown functions to work correctly. I haven't really tested that, though, because I don't have a system at hand where they are currently not working fine.

Another nice thing of it is that when running seat-inspect -v you get a dump of what logind/consolekit think about your system. I found it an interesting way to explore the new functionalities that we recently grew. The same can be done, and in more details, with loginctl calls, but I lacked a summary.

After writing this I feel a bit more competent, probably enough to sit at somebody's computer and poke into loginctl bits. I highly recommend the experience.

Posted Tue Feb 10 18:06:43 2015 Tags:

Playing with python, terminfo and command output

I am experimenting with showing progress on the terminal for a subcommand that is being run, showing what is happening without scrolling away the output of the main program, and I came out with this little toy. It shows the last X lines of a subcommand output, then gets rid of everything after the command has ended.

Usability-wise, it feels like a tease to me: it looks like I'm being shown all sorts of information then they are taken away from me before I managed to make sense of them. However, I find it cute enough to share:

#!/usr/bin/env python3
#coding: utf-8
# Copyright 2015 Enrico Zini <enrico@enricozini.org>.  Licensed under the terms
# of the GNU General Public License, version 2 or any later version.

import argparse
import fcntl
import select
import curses
import contextlib
import subprocess
import os
import sys
import collections
import shlex
import shutil
import logging

def stream_output(proc):
    """
    Take a subprocess.Popen object and generate its output, line by line,
    annotated with "stdout" or "stderr". At process termination it generates
    one last element: ("result", return_code) with the return code of the
    process.
    """
    fds = [proc.stdout, proc.stderr]
    bufs = [b"", b""]
    types = ["stdout", "stderr"]
    # Set both pipes as non-blocking
    for fd in fds:
        fcntl.fcntl(fd, fcntl.F_SETFL, os.O_NONBLOCK)
    # Multiplex stdout and stderr with different prefixes
    while len(fds) > 0:
        s = select.select(fds, (), ())
        for fd in s[0]:
            idx = fds.index(fd)
            buf = fd.read()
            if len(buf) == 0:
                fds.pop(idx)
                if len(bufs[idx]) != 0:
                    yield types[idx], bufs.pop(idx)
                types.pop(idx)
            else:
                bufs[idx] += buf
                lines = bufs[idx].split(b"\n")
                bufs[idx] = lines.pop()
                for l in lines:
                    yield types[idx], l
    res = proc.wait()
    yield "result", res

@contextlib.contextmanager
def miniscreen(has_fancyterm, name, maxlines=3, silent=False):
    """
    Show the output of a process scrolling in a portion of the screen.

    has_fancyterm: true if the terminal supports fancy features; if false, just
    write lines to standard output

    name: name of the process being run, to use as a header

    maxlines: maximum height of the miniscreen

    silent: do nothing whatsoever, used to disable this without needing to
            change the code structure

    Usage:
        with miniscreen(True, "my process", 5) as print_line:
            for i in range(10):
                print_line(("stdout", "stderr")[i % 2], "Line #{}".format(i))
    """
    if not silent and has_fancyterm:
        # Discover all the terminal control sequences that we need
        output_normal = str(curses.tigetstr("sgr0"), "ascii")
        output_up = str(curses.tigetstr("cuu1"), "ascii")
        output_clreol = str(curses.tigetstr("el"), "ascii")
        cols, lines = shutil.get_terminal_size()
        output_width = cols

        fg_color = (curses.tigetstr("setaf") or
                    curses.tigetstr("setf") or "")
        sys.stdout.write(str(curses.tparm(fg_color, 6), "ascii"))

        output_lines = collections.deque(maxlen=maxlines)

        def print_lines():
            """
            Print the lines in our buffer, then move back to the beginning
            """
            sys.stdout.write("{} progress:".format(name))
            sys.stdout.write(output_clreol)
            for msg in output_lines:
                sys.stdout.write("\n")
                sys.stdout.write(msg)
                sys.stdout.write(output_clreol)
            sys.stdout.write(output_up * len(output_lines))
            sys.stdout.write("\r")

        try:
            print_lines()

            def _progress_line(type, line):
                """
                Print a new line to the miniscreen
                """
                # Add the new line to our output buffer
                msg = "{} {}".format("." if type == "stdout" else "!", line)
                if len(msg) > output_width - 4:
                    msg = msg[:output_width - 4] + "..."
                output_lines.append(msg)
                # Update the miniscreen
                print_lines()

            yield _progress_line

            # Clear the miniscreen by filling our ring buffer with empty lines
            # then printing them out
            for i in range(maxlines):
                output_lines.append("")
            print_lines()
        finally:
            sys.stdout.write(output_normal)
    elif not silent:
        def _progress_line(type, line):
            print("{}: {}".format(type, line))
        yield _progress_line
    else:
        def _progress_line(type, line):
            pass
        yield _progress_line

def run_command_fancy(name, cmd, env=None, logfd=None, fancy=True, debug=False):
    quoted_cmd = " ".join(shlex.quote(x) for x in cmd)
    log.info("%s running command %s", name, quoted_cmd)
    if logfd: print("runcmd:", quoted_cmd, file=logfd)

    # Run the script itself on an empty environment, so that what was
    # documented is exactly what was run
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)

    with miniscreen(fancy, name, silent=debug) as progress:
        stderr = []
        for type, val in stream_output(proc):
            if type == "stdout":
                val = val.decode("utf-8")
                if logfd: print("stdout:", val, file=logfd)
                log.debug("%s stdout: %s", name, val)
                progress(type, val)
            elif type == "stderr":
                val = val.decode("utf-8")
                if logfd: print("stderr:", val, file=logfd)
                stderr.append(val)
                log.debug("%s stderr: %s", name, val)
                progress(type, val)
            elif type == "result":
                if logfd: print("retval:", val, file=logfd)
                log.debug("%s retval: %d", name, val)
                retval = val

    if retval != 0:
        lastlines = min(len(stderr), 5)
        log.error("%s exited with code %s", name, retval)
        log.error("Last %d lines of standard error:", lastlines)
        for line in stderr[-lastlines:]:
            log.error("%s: %s", name, line)

    return retval


parser = argparse.ArgumentParser(description="run a command showing only a portion of its output")
parser.add_argument("--logfile", action="store", help="specify a file where the full execution log will be written")
parser.add_argument("--debug", action="store_true", help="debugging output on the terminal")
parser.add_argument("--verbose", action="store_true", help="verbose output on the terminal")
parser.add_argument("command", nargs="*", help="command to run")
args = parser.parse_args()

if args.debug:
    loglevel = logging.DEBUG
elif args.verbose:
    loglevel = logging.INFO
else:
    loglevel = logging.WARN
logging.basicConfig(level=loglevel, stream=sys.stderr)
log = logging.getLogger()

fancy = False
if not args.debug and sys.stdout.isatty():
    curses.setupterm()
    if curses.tigetnum("colors") > 0:
        fancy = True

if args.logfile:
    logfd = open("output.log", "wt")
else:
    logfd = None

retval = run_command_fancy("miniscreen example", args.command, logfd=logfd)

sys.exit(retval)
Posted Wed Jan 21 11:13:31 2015 Tags:

Non importa che mi dai del voi

Dai, non importa che mi dai del voi

In che senso?

Eh, mi dici sempre "voi informatici", "voi tecnici", "voi..."

Posted Fri Dec 19 15:55:20 2014 Tags:
Posted Sat Jun 6 00:57:39 2009
osm

Pages about OpenStreetMap.

Computing time offsets between EXIF and GPS

I like the idea of matching photos to GPS traces. In Debian there is gpscorrelate but it's almost unusable to me because of bug #473362 and it has an awkward way of specifying time offsets.

Here at SoTM10 someone told me that exiftool gained -geosync and -geotag options. So it's just a matter of creating a little tool that shows a photo and asks you to type the GPS time you see in it.

Apparently there are no bindings or GIR files for gtkimageview in Debian, so I'll have to use C.

Here is a C prototype:

/*
 * gpsoffset - Compute EXIF time offset from a photo of a gps display
 *
 * Use with exiftool -geosync=... -geotag trace.gpx DIR
 *
 * Copyright (C) 2009--2010  Enrico Zini <enrico@enricozini.org>
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program; if not, write to the Free Software
 * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 */


#define _XOPEN_SOURCE /* glibc2 needs this */
#include <time.h>
#include <gtkimageview/gtkimageview.h>
#include <libexif/exif-data.h>
#include <stdio.h>
#include <stdlib.h>

static int load_time(const char* fname, struct tm* tm)
{
    ExifData* exif_data = exif_data_new_from_file(fname);
    ExifEntry* exif_time = exif_data_get_entry(exif_data, EXIF_TAG_DATE_TIME);
    if (exif_time == NULL)
    {
        fprintf(stderr, "Cannot find EXIF timetamp\n");
        return -1;
    }

    char buf[1024];
    exif_entry_get_value(exif_time, buf, 1024);
    //printf("val2: %s\n", exif_entry_get_value(t2, buf, 1024));

    if (strptime(buf, "%Y:%m:%d %H:%M:%S", tm) == NULL)
    {
        fprintf(stderr, "Cannot match EXIF timetamp\n");
        return -1;
    }

    return 0;
}

static time_t exif_ts;
static GtkWidget* res_lbl;

void date_entry_changed(GtkEditable *editable, gpointer user_data)
{
    const gchar* text = gtk_entry_get_text(GTK_ENTRY(editable));
    struct tm parsed;
    if (strptime(text, "%Y-%m-%d %H:%M:%S", &parsed) == NULL)
    {
        gtk_label_set_text(GTK_LABEL(res_lbl), "Please enter a date as YYYY-MM-DD HH:MM:SS");
    } else {
        time_t img_ts = mktime(&parsed);
        int c;
        int res;
        if (exif_ts < img_ts)
        {
            c = '+';
            res = img_ts - exif_ts;
        }
        else
        {
            c = '-';
            res = exif_ts - img_ts;
        }
        char buf[1024];
        if (res > 3600)
            snprintf(buf, 1024, "Result: %c%ds -geosync=%c%d:%02d:%02d",
                    c, res, c, res / 3600, (res / 60) % 60, res % 60);
        else if (res > 60)
            snprintf(buf, 1024, "Result: %c%ds -geosync=%c%02d:%02d",
                    c, res, c, (res / 60) % 60, res % 60);
        else 
            snprintf(buf, 1024, "Result: %c%ds -geosync=%c%d",
                    c, res, c, res);
        gtk_label_set_text(GTK_LABEL(res_lbl), buf);
    }
}

int main (int argc, char *argv[])
{
    // Work in UTC to avoid mktime applying DST or timezones
    setenv("TZ", "UTC");

    const char* filename = "/home/enrico/web-eddie/galleries/2010/04-05-Uppermill/P1080932.jpg";

    gtk_init (&argc, &argv);

    struct tm exif_time;
    if (load_time(filename, &exif_time) != 0)
        return 1;

    printf("EXIF time: %s\n", asctime(&exif_time));
    exif_ts = mktime(&exif_time);

    GtkWidget* window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
    GtkWidget* vb = gtk_vbox_new(FALSE, 0);
    GtkWidget* hb = gtk_hbox_new(FALSE, 0);
    GtkWidget* lbl = gtk_label_new("Timestamp:");
    GtkWidget* exif_lbl;
    {
        char buf[1024];
        strftime(buf, 1024, "EXIF time: %Y-%m-%d %H:%M:%S", &exif_time);
        exif_lbl = gtk_label_new(buf);
    }
    GtkWidget* date_ent = gtk_entry_new();
    res_lbl = gtk_label_new("Result:");
    GtkWidget* view = gtk_image_view_new();
    GdkPixbuf* pixbuf = gdk_pixbuf_new_from_file(filename, NULL);

    gtk_box_pack_start(GTK_BOX(hb), lbl, FALSE, TRUE, 0);
    gtk_box_pack_start(GTK_BOX(hb), date_ent, TRUE, TRUE, 0);

    gtk_signal_connect(GTK_OBJECT(date_ent), "changed", (GCallback)date_entry_changed, NULL);
    {
        char buf[1024];
        strftime(buf, 1024, "%Y-%m-%d %H:%M:%S", &exif_time);
        gtk_entry_set_text(GTK_ENTRY(date_ent), buf);
    }

    gtk_widget_set_size_request(view, 500, 400);
    gtk_image_view_set_pixbuf(GTK_IMAGE_VIEW(view), pixbuf, TRUE);
    gtk_container_add(GTK_CONTAINER(window), vb);
    gtk_box_pack_start(GTK_BOX(vb), view, TRUE, TRUE, 0);
    gtk_box_pack_start(GTK_BOX(vb), hb, FALSE, TRUE, 0);
    gtk_box_pack_start(GTK_BOX(vb), exif_lbl, FALSE, TRUE, 0);
    gtk_box_pack_start(GTK_BOX(vb), res_lbl, FALSE, TRUE, 0);
    gtk_widget_show_all(window);

    gtk_main ();

    return 0;
}

And here is its simple makefile:

CFLAGS=$(shell pkg-config --cflags gtkimageview libexif)
LDFLAGS=$(shell pkg-config --libs gtkimageview libexif)

gpsoffset: gpsoffset.c

It's a simple prototype but it's a working prototype and seems to do the job for me.

I currently cannot find out why after I click on the text box, there seems to be no way to give the focus back to the image viewer so I can control it with keys.

There is another nice algorithm to compute time offsets to be implemented: you choose a photo taken from a known place and drag it on that place on a map: you can then look for the nearest point on your GPX trace and compute the time offset from that.

I have seen that there are programs for geotagging photos that implement all such algorithms, and have a nice UI, but I haven't seen any in Debian.

Are there any such softwares that can be packaged?

If not, the interpolation and annotation tasks can now already be performed by exiftool, so it's just a matter of building a good UI, and I would love to see someone picking up the task.

Posted Sun Jul 11 12:34:04 2010 Tags:

Searching OSM nodes in Spatialite

Third step of my SoTM10 pet project: finding the POIs.

I put together a query to find all nodes with a given tag inside a bounding box, and also a query to find all the tag values for a given tag name inside a bounding box.

The result is this simple POI search engine:

#
# poisearch - simple geographical POI search engine
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#

from pysqlite2 import dbapi2 as sqlite

class PoiDB(object):
    def __init__(self):
        self.db = sqlite.connect("pois.db")
        self.db.enable_load_extension(True)
        self.db.execute("SELECT load_extension('libspatialite.so')")
        self.oldsearch = []
        self.bbox = None

    def set_bbox(self, xmin, xmax, ymin, ymax):
        '''Set bbox for searches'''
        self.bbox = (xmin, xmax, ymin, ymax)

    def tagid(self, name, val):
        '''Get the database ID for a tag'''
        c = self.db.cursor()
        c.execute("SELECT id FROM tag WHERE name=? AND value=?", (name, val))
        res = None
        for row in c:
            res = row[0]
        return res

    def tagnames(self):
        '''Get all tag names'''
        c = self.db.cursor()
        c.execute("SELECT DISTINCT name FROM tag ORDER BY name")
        for row in c:
            yield row[0]

    def tagvalues(self, name, use_bbox=False):
        '''
        Get all tag values for a given tag name,
        optionally in the current bounding box
        '''
        c = self.db.cursor()
        if self.bbox is None or not use_bbox:
            c.execute("SELECT DISTINCT value FROM tag WHERE name=? ORDER BY value", (name,))
        else:
            c.execute("SELECT DISTINCT tag.value FROM poi, poitag, tag"
                      " WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE ("
                      "       xmin >= ? AND xmax <= ? AND ymin >= ? AND ymax <= ?) )"
                      "   AND poitag.tag = tag.id AND poitag.poi = poi.id"
                      "   AND tag.name=?",
                      self.bbox + (name,))
        for row in c:
            yield row[0]

    def search(self, name, val):
        '''Get all name:val tags in the current bounding box'''
        # First resolve the tagid
        tagid = self.tagid(name, val)
        if tagid is None: return

        c = self.db.cursor()
        c.execute("SELECT poi.name, poi.data, X(poi.geom), Y(poi.geom) FROM poi, poitag"
                  " WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE ("
                  "       xmin >= ? AND xmax <= ? AND ymin >= ? AND ymax <= ?) )"
                  "   AND poitag.tag = ? AND poitag.poi = poi.id",
                  self.bbox + (tagid,))
        self.oldsearch = []
        for row in c:
            self.oldsearch.append(row)
            yield row[0], simplejson.loads(row[1]), row[2], row[3]

    def count(self, name, val):
        '''Count all name:val tags in the current bounding box'''
        # First resolve the tagid
        tagid = self.tagid(name, val)
        if tagid is None: return

        c = self.db.cursor()
        c.execute("SELECT COUNT(*) FROM poi, poitag"
                  " WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE ("
                  "       xmin >= ? AND xmax <= ? AND ymin >= ? AND ymax <= ?) )"
                  "   AND poitag.tag = ? AND poitag.poi = poi.id",
                  self.bbox + (tagid,))
        for row in c:
            return row[0]

    def replay(self):
        for row in self.oldsearch:
            yield row[0], simplejson.loads(row[1]), row[2], row[3]

Problem 3 solved: now on to the next step, building a user interface for it.

Posted Sat Jul 10 15:50:31 2010 Tags:

Importing OSM nodes into Spatialite

Second step of my SoTM10 pet project: creating a searchable database with the points. What a fantastic opportunity to learn Spatialite.

Learning Spatialite is easy. For example, you can use the two tutorials with catchy titles that assume your best wish in life is to create databases out of shapefiles using a pre-built, i386-only executable GUI binary downloaded over an insecure HTTP connection.

To be fair, the second of those tutorials is called "An almost Idiot's Guide", thus expliciting the requirement of being an almost idiot in order to happily acquire and run software in that way.

Alternatively, you can use A quick tutorial to SpatiaLite which is so quick it has examples that lead you to write SQL queries that trigger all sorts of vague exceptions at insert time. But at least it brought me a long way forward, at which point I could just cross reference things with PostGIS documentation to find out the right way of doing things.

So, here's the importer script, which will probably become my reference example for how to get started with Spatialite, and how to use Spatialite from Python:

#!/usr/bin/python

#
# poiimport - import nodes from OSM into a spatialite DB
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#

import xml.sax
import xml.sax.handler
from pysqlite2 import dbapi2 as sqlite
import simplejson
import sys
import os

class OSMPOIReader(xml.sax.handler.ContentHandler):
    '''
    Filter SAX events in a OSM XML file to keep only nodes with names
    '''
    def __init__(self, consumer):
        self.consumer = consumer

    def startElement(self, name, attrs):
        if name == "node":
            self.attrs = attrs
            self.tags = dict()
        elif name == "tag":
            self.tags[attrs["k"]] = attrs["v"]

    def endElement(self, name):
        if name == "node":
            lat = float(self.attrs["lat"])
            lon = float(self.attrs["lon"])
            id = int(self.attrs["id"])
            #dt = parse(self.attrs["timestamp"])
            uid = self.attrs.get("uid", None)
            uid = int(uid) if uid is not None else None
            user = self.attrs.get("user", None)

            self.consumer(lat, lon, id, self.tags, user=user, uid=uid)

class Importer(object):
    '''
    Create the spatialite database and populate it
    '''
    TAG_WHITELIST = set(["amenity", "shop", "tourism", "place"])

    def __init__(self, filename):
        self.db = sqlite.connect(filename)
        self.db.enable_load_extension(True)
        self.db.execute("SELECT load_extension('libspatialite.so')")
        self.db.execute("SELECT InitSpatialMetaData()")
        self.db.execute("INSERT INTO spatial_ref_sys (srid, auth_name, auth_srid,"
                        " ref_sys_name, proj4text) VALUES (4326, 'epsg', 4326,"
                        " 'WGS 84', '+proj=longlat +ellps=WGS84 +datum=WGS84"
                        " +no_defs')")
        self.db.execute("CREATE TABLE poi (id int not null unique primary key,"
                        " name char, data text)")
        self.db.execute("SELECT AddGeometryColumn('poi', 'geom', 4326, 'POINT', 2)")
        self.db.execute("SELECT CreateSpatialIndex('poi', 'geom')")
        self.db.execute("CREATE TABLE tag (id integer primary key autoincrement,"
                        " name char, value char)")
        self.db.execute("CREATE UNIQUE INDEX tagidx ON tag (name, value)")
        self.db.execute("CREATE TABLE poitag (poi int not null, tag int not null)")
        self.db.execute("CREATE UNIQUE INDEX poitagidx ON poitag (poi, tag)")
        self.tagid_cache = dict()

    def tagid(self, k, v):
        key = (k, v)
        res = self.tagid_cache.get(key, None)
        if res is None:
            c = self.db.cursor()
            c.execute("SELECT id FROM tag WHERE name=? AND value=?", key)
            for row in c:
                self.tagid_cache[key] = row[0]
                return row[0]
            self.db.execute("INSERT INTO tag (id, name, value) VALUES (NULL, ?, ?)", key)
            c.execute("SELECT last_insert_rowid()")
            for row in c:
                res = row[0]
            self.tagid_cache[key] = res
        return res

    def __call__(self, lat, lon, id, tags, user=None, uid=None):
        # Acquire tag IDs
        tagids = []
        for k, v in tags.iteritems():
            if k not in self.TAG_WHITELIST: continue
            for val in v.split(";"):
                tagids.append(self.tagid(k, val))

        # Skip elements that don't have the tags we want
        if not tagids: return

        geom = "POINT(%f %f)" % (lon, lat)
        self.db.execute("INSERT INTO poi (id, geom, name, data)"
                        "     VALUES (?, GeomFromText(?, 4326), ?, ?)", 
                (id, geom, tags["name"], simplejson.dumps(tags)))

        for tid in tagids:
            self.db.execute("INSERT INTO poitag (poi, tag) VALUES (?, ?)", (id, tid))


    def done(self):
        self.db.commit()

# Get the output file name
filename = sys.argv[1]

# Ensure we start from scratch
if os.path.exists(filename):
    print >>sys.stderr, filename, "already exists"
    sys.exit(1)

# Import
parser = xml.sax.make_parser()
importer = Importer(filename)
handler = OSMPOIReader(importer)
parser.setContentHandler(handler)
parser.parse(sys.stdin)
importer.done()

Let's run it:

$ ./poiimport pois.db < pois.osm 
SpatiaLite version ..: 2.4.0    Supported Extensions:
        - 'VirtualShape'        [direct Shapefile access]
        - 'VirtualDbf'          [direct Dbf access]
        - 'VirtualText'         [direct CSV/TXT access]
        - 'VirtualNetwork'      [Dijkstra shortest path]
        - 'RTree'               [Spatial Index - R*Tree]
        - 'MbrCache'            [Spatial Index - MBR cache]
        - 'VirtualFDO'          [FDO-OGR interoperability]
        - 'SpatiaLite'          [Spatial SQL - OGC]
PROJ.4 Rel. 4.7.1, 23 September 2009
GEOS version 3.2.0-CAPI-1.6.0
$ ls -l --si pois*
-rw-r--r-- 1 enrico enrico 17M Jul  9 23:44 pois.db
-rw-r--r-- 1 enrico enrico 37M Jul  9 16:20 pois.osm
$ spatialite pois.db
SpatiaLite version ..: 2.4.0    Supported Extensions:
        - 'VirtualShape'        [direct Shapefile access]
        - 'VirtualDbf'          [direct DBF access]
        - 'VirtualText'         [direct CSV/TXT access]
        - 'VirtualNetwork'      [Dijkstra shortest path]
        - 'RTree'               [Spatial Index - R*Tree]
        - 'MbrCache'            [Spatial Index - MBR cache]
        - 'VirtualFDO'          [FDO-OGR interoperability]
        - 'SpatiaLite'          [Spatial SQL - OGC]
PROJ.4 version ......: Rel. 4.7.1, 23 September 2009
GEOS version ........: 3.2.0-CAPI-1.6.0
SQLite version ......: 3.6.23.1
Enter ".help" for instructions
spatialite> select id from tag where name="amenity" and value="fountain";
24
spatialite> SELECT poi.name, poi.data, X(poi.geom), Y(poi.geom) FROM poi, poitag WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE (xmin >= 2.56 AND xmax <= 2.90 AND ymin >= 41.84 AND ymax <= 42.00) ) AND poitag.tag = 24 AND poitag.poi = poi.id;
Font Picant de la Cellera|{"amenity": "fountain", "name": "Font Picant de la Cellera"}|2.616045|41.952449
Font de Can Pla|{"amenity": "fountain", "name": "Font de Can Pla"}|2.622354|41.974724
Font de Can Ribes|{"amenity": "fountain", "name": "Font de Can Ribes"}|2.62311|41.979193

It's impressive: I've got all sort of useful information for the whole of Spain in just 17Mb!

Let's put it to practice: I'm thirsty, is there any water fountain nearby?

spatialite> SELECT count(1) FROM poi, poitag WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE (xmin >= 2.80 AND xmax <= 2.85 AND ymin >= 41.97 AND ymax <= 42.00) ) AND poitag.tag = 24 AND poitag.poi = poi.id;
0

Ouch! No water fountains mapped in Girona... yet.

Problem 2 solved: now on to the next step, trying to show the results in some usable way.

Posted Sat Jul 10 09:10:35 2010 Tags:

Filtering nodes out of OSM files

I have a pet project here at SoTM10: create a tool for searching nearby POIs while offline.

The idea is to have something in my pocket (FreeRunner or N900), which doesn't require an internet connection, and which can point me at the nearest fountains, post offices, atm machines, bars and so on.

The first step is to obtain a list of POIs.

In theory one can use Xapi but all the known Xapi servers appear to be down at the moment.

Another attempt is to obtain it by filtering all nodes with the tags we want out of a planet OSM extract. I downloaded the Spanish one and set to work.

First I tried with xmlstarlet, but it ate all the RAM and crashed my laptop, because for some reason, on my laptop the Linux kernels up to 2.6.32 (don't now about later ones) like to swap out ALL running apps to cache I/O operations, which mean that heavy I/O operations swap out the very programs performing them, so the system gets caught in some infinite I/O loop and dies. Or at least this is what I've figured out so far.

So, we need SAX. I put together this prototype in Python, which can process a nice 8MB/s of OSM data for quite some time with a constant, low RAM usage:

#!/usr/bin/python

#
# poifilter - extract interesting nodes from OSM XML files
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#


import xml.sax
import xml.sax.handler
import xml.sax.saxutils
import sys

class XMLSAXFilter(xml.sax.handler.ContentHandler):
    '''
    A SAX filter that is a ContentHandler.

    There is xml.sax.saxutils.XMLFilterBase in the standard library but it is
    undocumented, and most of the examples using it you find online are wrong.
    You can look at its source code, and at that point you find out that it is
    an offensive practical joke.
    '''
    def __init__(self, downstream):
        self.downstream = downstream

    # ContentHandler methods

    def setDocumentLocator(self, locator):
        self.downstream.setDocumentLocator(locator)

    def startDocument(self):
        self.downstream.startDocument()

    def endDocument(self):
        self.downstream.endDocument()

    def startPrefixMapping(self, prefix, uri):
        self.downstream.startPrefixMapping(prefix, uri)

    def endPrefixMapping(self, prefix):
        self.downstream.endPrefixMapping(prefix)

    def startElement(self, name, attrs):
        self.downstream.startElement(name, attrs)

    def endElement(self, name):
        self.downstream.endElement(name)

    def startElementNS(self, name, qname, attrs):
        self.downstream.startElementNS(name, qname, attrs)

    def endElementNS(self, name, qname):
        self.downstream.endElementNS(name, qname)

    def characters(self, content):
        self.downstream.characters(content)

    def ignorableWhitespace(self, chars):
        self.downstream.ignorableWhitespace(chars)

    def processingInstruction(self, target, data):
        self.downstream.processingInstruction(target, data)

    def skippedEntity(self, name):
        self.downstream.skippedEntity(name)

class OSMPOIHandler(XMLSAXFilter):
    '''
    Filter SAX events in a OSM XML file to keep only nodes with names
    '''
    PASSTHROUGH = ["osm", "bound"]
    TAG_WHITELIST = set(["amenity", "shop", "tourism", "place"])

    def startElement(self, name, attrs):
        if name in self.PASSTHROUGH:
            self.downstream.startElement(name, attrs)
        elif name == "node":
            self.attrs = attrs
            self.tags = []
            self.propagate = False
        elif name == "tag":
            if self.tags is not None:
                self.tags.append(attrs)
                if attrs["k"] in self.TAG_WHITELIST:
                    self.propagate = True
        else:
            self.tags = None
            self.attrs = None

    def endElement(self, name):
        if name in self.PASSTHROUGH:
            self.downstream.endElement(name)
        elif name == "node":
            if self.propagate:
                self.downstream.startElement("node", self.attrs)
                for attrs in self.tags:
                    self.downstream.startElement("tag", attrs)
                    self.downstream.endElement("tag")
                self.downstream.endElement("node")

    def ignorableWhitespace(self, chars):
        pass

    def characters(self, content):
        pass

# Simple stdin->stdout XMl filter
parser = xml.sax.make_parser()
handler = OSMPOIHandler(xml.sax.saxutils.XMLGenerator(sys.stdout, "utf-8"))
parser.setContentHandler(handler)
parser.parse(sys.stdin)

Let's run it:

$ bzcat /store/osm/spain.osm.bz2 | pv | ./poifilter > pois.osm
[...]
$ ls -l --si pois.osm
-rw-r--r-- 1 enrico enrico 19M Jul 10 23:56 pois.osm
$ xmlstarlet val pois.osm 
pois.osm - valid

Problem 1 solved: now on to the next step: importing the nodes in a database.

Posted Fri Jul 9 16:28:15 2010 Tags:

Mapping using the Openmoko FreeRunner headset

The FreeRunner has a headset which includes a microphone and a button. When doing OpenStreetMap mapping, it would be very useful to be able to keep tangogps on the display and be able to mark waypoints using the headset button, and to record an audio track using the headset microphone.

In this way, I can use tangogps to see where I need to go, where it's already mapped and where it isn't, and then I can use the headset to mark waypoints corresponding to the audio track, so that later I can take advantage of JOSM's audio mapping features.

Enter audiomap:

$ audiomap --help
Usage: audiomap [options]

Create a GPX and audio trackFind the times in the wav file when there is clear
voice among the noise

Options:
  --version      show program's version number and exit
  -h, --help     show this help message and exit
  -v, --verbose  verbose mode
  -m, --monitor  only keep the GPS on and monitor satellite status
  -l, --levels   only show input levels

If called without parameters, or with -v which is suggested, it will:

  1. Fix the mixer settings so that it can record from the headset and detect headset button presses.
  2. Show a monitor of GPS satellite information until it gets a fix.
  3. Synchronize the system time with the GPS time so that the timestamps of the files that are created afterwards are accurate.
  4. Start recording a GPX track.
  5. Start recording audio.
  6. Record a GPX waypoint for every headset button press.

When you are done, you stop audiomap with ^C and it will properly close the .wav file, close the tags in the GPX waypoint and track files and restore the mixer settings.

You can plug the headset out and record using the handset microphone, but then you will not be able to set waypoints until you plug the headset back in.

After you stop audiomap, you will have a track, waypoints and .wav file ready to be loaded in JOSM.

Big thanks go to Luca Capello for finding out how to detect headset button presses.

Posted Sun Jun 7 23:51:37 2009 Tags:

Simple tool to query the GPS using the OpenMoko FSO stack

I was missing a simple command line tool that allows me to perform basic GPS queries in shellscripts.

Enter getgps:

# getgps --help
Usage: getgps [options]

Simple GPS query tool for the FSO stack

Options:
  --version          show program's version number and exit
  -h, --help         show this help message and exit
  -v, --verbose      verbose mode
  -q, --quiet        suppress normal output
  --fix              check if we have a fix
  -s, --sync-time    set system time from GPS time
  --info             get all GPS information
  --info-connection  get GPS connection information
  --info-fix         get GPS fix information
  --info-position    get GPS position information
  --info-accuracy    get GPS accuracy information
  --info-course      get GPS course information
  --info-time        get GPS time information
  --info-satellite   get GPS satellite information

So finally I can write little GPS-aware scripts:

if getgps --fix -q
then
    start_gps_aware_program
else
    start_gps_normal_program
fi

Or this.

Posted Sun Jun 7 17:59:32 2009 Tags:

Voice-controlled waypoints

I have it in my TODO list to implement taking waypoints when pressing the headset button of the openmoko, but that is not done yet.

In the meantime, I did some experiments with audio mapping, and since I did not manage to enter waypoints while recording them, I was looking for a way to make use of them anyway.

Enter findvoice:

$ ./findvoice  --help
Usage: findvoice [options] wavfile

Find the times in the wav file when there is clear voice among the noise

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -v, --verbose         verbose mode
  -p NUM, --percentile=NUM
            percentile to use to discriminate noise from voice
            (default: 90)
  -t, --timestamps      print timestamps instead of human readable information

You give it a wav file, and it will output a list of timestamps corresponding to where it things that you were talking clearly and near the FreeRunner / voice recorder instead of leaving the recorder dangling to pick up background noise.

Its algorithm is crude and improvised because I have no background whatsoever in audio processing, but it basically finds those parts of the audio file where the variance of the samples is above a given percentile: the higher the percentile, the less timestamps you get; the lower the percentile, the more likely it is that it picks a period of louder noise.

For example, you can automatically extract waypoints out of an audio file by using it together with gpxinterpolate:

./findvoice -t today.wav | ./gpxinterpolate today.gpx > today-waypoints.gpx

The timestamps it outputs are computed using the modification time of the .wav file: if your system clock was decently synchronised (which you can do with getgps), then the mtime of the wav is the time of the end of the recording, which gives the needed reference to compute timestamps that are absolute in time.

For example:

getgps --sync-time
arecord file.wav
^C
./findvoice -t file.wav | ./gpxinterpolate today.gpx > today-waypoints.gpx
Posted Sun Jun 7 02:48:40 2009 Tags:

Geocoding Unix timestamps

Geocoding EXIF tags in JPEG images is fun, but there is more that can benefit from interpolating timestamps over a GPX track.

Enter gpxinterpolate:

$ ./gpxinterpolate --help
Usage: gpxinterpolate [options] gpxfile [gpxfile...]

Read one or more GPX files and a list of timestamps on standard input. Output
a GPX file with waypoints at the location of the GPX track at the given
timestamps.

Options:
  --version      show program's version number and exit
  -h, --help     show this help message and exit
  -v, --verbose  verbose mode

For example, you can create waypoints interpolating file modification times:

find . -printf "%Ts %p\n" | ./gpxinterpolate ~/tracks/*.gpx > myfiles.gpx

In case you wonder where you were when you modified or accessed a file, now you can find out.

Posted Sun Jun 7 02:07:43 2009 Tags:

Recording audio on the FreeRunner

The FreeRunner can record audio. It is nice to record audio: for example I can run the recording in background while I keep tangogps in the screen, and take audio notes about where I am while I am doing mapping for OpenStreetMap.

Here is the script that I put together to create geocoded audio notes:

#!/bin/sh

WORKDIR=~/rec
TMPINFO=`mktemp $WORKDIR/info.XXXXXXXX`

# Sync system time and get GPS info
echo "Synchronising system time..."
getgps --sync-time --info > $TMPINFO

# Compute an accurate basename for the files we generate
BASENAME=~/rec/rec-$(date +%Y-%m-%d-%H-%M-%S)
# Then give a proper name to the file with saved info
mv $TMPINFO $BASENAME.info

# Proper mixer settings for recording
echo "Recording..."
alsactl -f /usr/share/openmoko/scenarios/voip-handset.state restore
arecord -D hw -f cd -r 8000 -t wav $BASENAME.wav

echo "Done"

It works like this:

  1. It synchronizes the system time from the GPS (if there is a fix) so that the timestamps on the wav files will be as accurate as possible.
  2. It also gets all sort of information from the GPS and stores them into a file, should you want to inspect it later.
  3. It records audio until it gets interrupted.

The file name of the files that it generates corresponds to the beginning of the recording. The mtime of the wav file obviously corresponds to the end of the recording. This can be used to later georeference the start and end point of the recording.

You can use this to check mixer levels and that you're actually getting any input:

arecord -D hw -f cd -r 8000 -t wav -V mono /dev/null

The getgps script is now described in its own post.

You may now want to experiment, in JOSM, with "Preferences / Audio settings / Modified times (time stamps) of audio files".

Posted Sun Jun 7 01:30:37 2009 Tags:

Playing with a Holux M-241

Here is how to download tracks from the Holux M-241:

gpsbabel -t -r -w -i m241 -f /dev/ttyUSB0 -o gpx -F `date +'%Y-%m-%d-%H%M%S'`.gpx

It might work also via bluetooth, but I have not tried yet.

Now, until M-241 support will be released in a stable version of gpsbabel, here is how to compile the version from CVS.

Get the sources:

cvs -d:pserver:anonymous@gpsbabel.cvs.sourceforge.net:/cvsroot/gpsbabel login
cvs -z3 -d:pserver:anonymous@gpsbabel.cvs.sourceforge.net:/cvsroot/gpsbabel co -P gpsbabel

Untar this to debianise the sources.

If it does to compile because of some errors in lmx.c, apply this patch.

Then you can install the resulting package and (hopefully) be happy.

Note, after downloading the logs, gpsbabel currently turns on logging. Here is a patch to disable that behaviour.

Finally, if you want to hack around a little on the unit, you can play with mtkbabel: the source code is simple, and most of the MTK protocol is implemented, so you can easily feed your own commands to the MTK. Documentation about the commands can be found here:

The links are taken from a post in the GPSPasSion forum.

For a source of ideas of what commands you can send, you can look into the source code of BT747. For example, to set the M-241 to 2Hz fix, you can add this to mtkbabel:

packet_send('PMTK300,500,0,0,0.0,0.0');
$ret = packet_wait('PMTK001,300,3');

And if you want to set the MTK to an insane 5Hz fix rate, to take really fine grained gpx traces with your laptop, you can use this:

packet_send('PMTK300,200,0,0,0.0,0.0');
$ret = packet_wait('PMTK001,300,3');

Don't forget to set the serial speed to 38400 before talking with the unit:

stty 38400 < /dev/ttyUSB0
Posted Sat Jun 6 00:57:39 2009 Tags:
Posted Sat Jun 6 00:57:39 2009

Pages about Debtags.

Evolution's old odd mail folders to mbox

Something wrong happened in my dad's Evolution. It just would get stuck checking mail forever, with no useful diagnostic that I could find. Fun. Not.

Anyway, I solved by resetting everything to factory defaults, moving away all gconf entries and .evolution/ files. Then it started to work again, of course then I needed to reconfigure it from scratch.

It turned out however that some old mail was only archived locally, and in a kind of weird format that looks like this:

$ ls -la Enrico/
total 336
drwx------ 2 enrico enrico   4096 Jul 23 03:05 .
drwxr-xr-x 7 enrico enrico   4096 Jul 23 03:12 ..
-rw------- 1 enrico enrico   3230 Dec  4  2010 113.HEADER
-rw------- 1 enrico enrico  14521 Dec  4  2010 113.TEXT
-rw------- 1 enrico enrico   3209 Oct 22  2010 134.HEADER
-rw------- 1 enrico enrico   2937 Oct 22  2010 134.TEXT
-rw------- 1 enrico enrico   3116 Jun 27  2011 15.
-rw------- 1 enrico enrico   3678 Jun 27  2011 168.
-rw------- 1 enrico enrico     73 Apr 27  2009 22.1.MIME
-rw------- 1 enrico enrico   3199 Apr 27  2009 22.2
-rw------- 1 enrico enrico     88 Apr 27  2009 22.2.MIME
[...]

I couldn't even find the name of that mail folder layout, let alone conversion tools. So I had to sit down and waste my sunday break writing software to convert that to a mbox file. Here's the tool, may it save you the awful time I had today: http://anonscm.debian.org/gitweb/?p=users/enrico/evo2mbox.git

Note: feel free to fork it, or send patches, but don't bother with feature requests. Evolution isn't and won't be a personal interest of mine. Anything that makes an afternoon at my parents more tiresome than a whole busy month of paid work, doesn't deserve to be.

Luckily they now seem to have changed the local folder format to Maildir.

Posted Mon Jul 23 03:27:50 2012 Tags:

Giving away distromatch

at last year's Fosdem I tried to inject a lot of energy into distromatch but shortly afterwards I've had to urgently rewrite the nm.debian.org website.

After Lars Wirzenius GTDFH talks in Bologna and Varese I wrote a tool which, among other things, is able to scan my home dir and list how many projects I'm working on.

The output was scary. Like, they are too many. Like, I couldn't even recite the list out of memory. And since I couldn't do that, I had no idea there were so many. And I kept being stressful because I couldn't manage to take care of them all properly.

Now that I became conscious of the situation, it's time to deal with it like a grown up, and politely back off from some of my irresponsible responsibilities.

Distromatch is one of them. It had just started as a proof of concept prototype, and I had the vision that it could be the basis for a fantastic culture of sharing and exchange of information across distributions.

I need to distinguish the vision from the responsibility. I still have that vision for distromatch, but I cannot take responsibility for making it happen.

So I am giving it up to anyone who has the time and resources to pick up that responsibility.

Current status

It works well enough as a prototype. I believe it can successfully map a large enough slice of packages, that one can prototype stuff based on it.

I have for example used it to export the Debtags categories for other distros, and the resulting file looked big enough to be used for prototyping category-based features on distributions that don't have them yet.

I think it also works well enough to support a few common use cases, like sharing screenshots, or doing most of the work of converting dependency lists from a distro to another.

And finally, anyone can deploy it, and work on it.

Existing data sources

Everything I index in the Debian distromatch deployment is available at http://dde.debian.net/exports/distromatch/. The rpm-based data in there comes from an export script I wrote that runs on Sophie, but which I cannot maintain properly.

This is an experimental export of Fedora and OpenSUSE data: http://tmp.vuntz.net/misc/distromatch/distromatch-opensuse-fedora.tar

All existing export scripts are found in distromatch git repo on gitorious.

Contacts I gathered at Fosdem

At Fosdem I devoted quite some work to get contacts from all possible distributions and software repositories, so that distromatch could be hooked into them. Here is a dump of what I have collected:

  • Debian: me
  • OpenSuse: Vincent Untz and Adrian Schröter
  • Fedora: Tom "Spot" Callaway
  • Arch: Tasser on IRC
  • CPAN: contact the people of https://metacpan.org/, on irc.perl.org:#metacpan or make an issue on github
  • NetBSD: ask on #netbsd on Freenode
  • FreeBSD: Baptiste Daroussin (bapt)
  • Mageia: Olivier Thauvin

Some of those contacts may have "expired" in the meantime: I wouldn't assume all of them still remember talking with me, although most probably still do.

My commitment for the time being

I am happy to commit, at the moment, to maintaining a working data export for Debian data. I can take responsibility for making it so that the Debian data for it stays up to date, and to fix it asap if it isn't the case.

I hope that now someone can take distromatch over from me, and make it grow to achieve its great potential.

Posted Sat Jul 21 16:54:18 2012 Tags:

More diversity in Debian skills

This blog post has been co-authored with Francesca Ciceri.

In his Debconf talk, zack said:

We need to understand how to invite people with different backgrounds than packaging to join the Debian project [...] I don't know what exactly, but we need to do more to attract those kinds of people.

Francesca and I know what we could do: make other kinds of contributions visible.

Basically, we should track and acknowledge the contributions of webmasters, translators, programmers, sysadmins, event organisers, and so on, at the same level as what we do for packagers: DDPO, minechangelogs, Portfolio...

For any non-packaging activity that we can make visible and credited, we get:

  • to acknowledge the people who do it, and show that they are active contributors in the project;

  • to acknowledge the work that gets done, and show the actual amount of non-packaging work that gets done in Debian every day;

  • to allow non-packagers to have a reputation, too: first of all, they deserve it, and among other things, it would make nm processing trivial.  

Here's an example: who's the lead translator for German? And if you are German, who's the lead translator for Spanish? Czech? Thai? I (Enrico) don't know the answers, not even for Italian, but we all should! Or at least it should be trivial to find out.

To start to change this, is just a matter of programming.

Francesca already worked on a list of trackable data sources, at least for translators.

Here are some more details, related to translation:

  • Translations can be tracked via the i18n robot (and relative statistics). This works only with teams who activated the robot and actively use the pseudo-urls in their messages on localisation mailing lists. Some translators don't bother to do it but it's ok to only support the main workflow. It beats extracting .po files from l10n-tagged BTS bugs at any rate.

  • DPN and website translations: for wml pages there's a specific field to be extracted for each translated page: grep for maintainer="name" on normal wml pages, while for DPN translations we have a specific translator="name" field. The problem is that this field is not mandatory, so sometimes there's no indication of the maintainer. Again, it's ok to only support the main workflow.

    Anyway, this is preferable to the cvs log: often the commit is done by the coordinator of the team and not by the actual translator. See above for the alternative solution of using the statistics provided by the i18n bot.

  • DDTSS: since the new release of DDTSS-Django, done by Martijn van Oosterhout about a year ago, the contributions are by default non-anonymous. This should be easy to track.

  • http://wiki.debian.org: it is more complicated because in the wiki we do not have a proper l10n translation workflow, so the only thing that can be tracked are changelogs $LANG/* pages. A nice idea would be to have translated pages list the version of the page that was translated and who did the translation.

  • translation of debian manuals and release notes: usually in the translation of manuals and long documentation there is a specific translator field.

And here are some notes about other fields:

  • DPN editors: for each issue there's a list of editors at the bottom of the page. In the wml: grep for editor=.

  • Artwork: artwork submitted via debianart are easy to track on the portal. Anyway usually you can find the author in the license and copyright file.

  • Programming: the only thing we have is the list of services which can be expanded if needed.

  • Press and publicity: there seems to be not much besides svn logs.

  • l10n-english: The Smith Review Project page has some tracking links. Other activities can probably only be tracked, at the moment, via mailing list activity.

  • Events: we can use the "main coordinator" field on www.debian.org/events/$year/$date-$eventname.wml: grep for <define-tag coord>; for events not published on the http://www.debian.org, but only on http://wiki.debian.org, the coordinator or the contact for the event is usually present on the page itself.

  • Sysadmins: we haven't asked DSA.

And finally, if you are still wondering who those translation coordinators are, they are listed here, although not all teams keep that page up to date.

Of course, when a data source is too hard to mine, it can make sense to see if the workflow could be improved, rather than spending months writing compicated mining code.

This is a fun project for people at Debconf to get together and try.

If by the end of the conference we had a way to credit some group of non-packaging contributors, even if just one like translators or website contributors, at least we would finally have started having official trackers for the activities of non-packagers.

Posted Thu Jul 12 14:01:54 2012 Tags:

Debtags for derivative distributions

Sometimes I do cool stuff and I forget to announce it.

Ok, so I recently announced a new Debtags website.

I forgot to say in the announcement that the new website does not only know of Debian packages: see for example this page, at the very bottom it says: "Distributions: oneiric, precise, sid, testing".

This means that already, here and now, debtags.debian.net can be used to tag packages from both Debian and Ubuntu, and can easily be extended to cover the entire Debian ecosystem.

If you are a package maintainer, you will notice that your maintainer page shows your packages from everywhere. If you want to filter things a bit, for example hide obsolete packages from an old Debian Stable or Ubuntu LTS, just click on the "Settings" link on the top right to configure the page.

How it works

The magic is in this mergepackages script, which is run daily, and exports merged Packages files at dde.debian.net. The debtags.debian.net concept of Packages and Sources files are just those all-merged.gz and all-merged-sources.gz.

The merging is simple: that rebuild script processes files in order, and the first version of a package that is found is chosen as the base for the one that will go in the merged Packages file. Some fields like "Description" are just taken from this pivot package, others like Architecture or dependencies are merged into it. It's arbitrary, but works for me: the result has all the packages with all their possible architectures and dependencies, and is ready to be indexed with apt-xapian-index.

At the moment I pull data from Debian and Ubuntu, but you can see that the script can easily be extended to pull data from any Debian-style ftp archive, so any Debian derivative can go in. I've already started negotiations with the Derivatives Census on how to add any Debian derivative and keep the list up to date.

How to export tags for your own distribution

I'll use Ubuntu as an example since the data is already available.

The way you add Debtags to the Ubuntu packages file is just this one:

  1. Get the full reviewed tag database
  2. Optionally filter out those packages that you are not interested in
  3. Tweak this script to build an overrides file.
  4. Give the overrides file to your favourite ftp archive building tool.

The make-overrides is a bit rusty: if you improve it, please send me your changes.

That is it, nothing else required, no excuses, it's ready, here, now!

Hitches and gotchas

This merged Packages file is a bit of a hack, and suffers from name conflicts across distributions, where two different softwares are packaged in two different distributions with the same name.

Ideally, name conflicts should not happen: if a derivative decided to package kate and call it gedit, they deserve to have it tagged uitoolkit::gtk. I think it's rather important that the whole Debian ecosystem works as much as possible with a single package namespace.

However, that reasoning fails if you take time into account: packages get renamed, like git and chromium, and may mean completely different things, for example, if you compare Debian Stable with Debian Sid.

This last is a problem caused by debtags only working with package names but not package versions. I have a strategy in mind based on being able to override the stable tag database using headers in debian/control; it still needs some details sorted out, but I'm confident we will be able to address these issues properly soon enough.

Why stop at the Debian ecosystem?

Why indeed. I'm clearly trying to use FOSDEM, and the CrossDistribution devroom as the venue to discuss just that.

Posted Fri Jan 20 15:12:33 2012 Tags:

Deploying distromatch

I have been working on allowing anyone to set up their own distromatch instance.

For Debian and Ubuntu, I can easily generate the distromatch input using UDD and the Contents files found in any mirrors.

For the whole RPM world, thanks to Olivier Thauvin I have been able to set up regular exports from the vast Sophie database.

I have set up distromatch access on DDE, which can also serve as a list of all working distributions so far. If you have access to the full dataset of package names and package contents for a distribution not in that list, please get in touch and we can add it.

I'm also exporting the full raw dataset which enables anyone to set up the same distromatch environment on their own machines.

Here is how:

# Get distromatch
git clone git://gitorious.org/appstream/distromatch.git
cd distromatch

# Fetch distribution information (updated every 2 days)
wget http://dde.debian.net/exports/distromatch-all.tar.gz

# Unpack it
mkdir data
tar -C data -zxf distromatch-all.tar.gz

# Reindex it (use --verbose if you are curious)
./distromatch --datadir=data --reindex --verbose

# Run it
./distromatch --datadir=data debian gedit

What does this mean? For example it means that if another distribution has some data (categories, screenshots...) that your distribution doesn't have, you can use distromatch to translate package names, then go and get it!

My next step is going to be to improve the distromatch functionality in DDE and possibly build a simple user friendly web interface to it. If you have some JQuery experience and would like to help, don't wait to get in touch.

Posted Fri Feb 18 13:46:30 2011 Tags:

update-apt-xapian-index on other distros

I've drafted a little HOWTO on using apt-xapian-index on non-Debian distributions.

The procedure has been tried on Mageia with some success, and there's no reason it wouldn't work everywhere else: the index itself does not depend on anything distro-specific.

Posted Tue Jan 25 23:01:45 2011 Tags:

Cross-distro Meeting on Application Installer

I have been to a Cross-distro Meeting on Application Installer which to the best of our knowledge is also the first one of its kind. Credit goes to Vincent Untz for organising it, to OpenSUSE for hosting it and to the various sponsors for getting us there.

It went surprisingly well. We got along, got stuff done, did as much work as possible to agree on as many formats, protocols and technologies as we possibly could.

The timing of it is very important, as most major distros would like to adopt some of the features that just became popular in the various new app markets and stores, such as screenshots, user comments and ratings. It looks like a lot of new code is about to be written, or a lot of existing code is about to gain quite a bit of popularity.

For my part, I presented the work on Debtags and apt-xapian-index.

With regards to Debtags, other distros seem to be missing a compehensive classification system, and Debtags is, well, it.

With regards to apt-xapian-index, we just noticed that it's the perfect back-end for what everyone would like to do, and the index structure is rather distribution-agnostic, and it's been road-tested with considerable success by at least software-center, so it attracted quite a bit of interest, and will likely attract some more.

Just to prove a point I put together a prototype webby markety appy thing in just a few hours of work.

The meeting was also the ideal place to create a joint effort to match package names across distributions, which means that a lot of things that were hard to share before, such as screenshots, tags and patches, are suddenly not hard to share anymore.

Posted Sat Jan 22 01:40:50 2011 Tags:

A prototype webby markety appy thing

What better way to introduce my work at an Application Installer meeting than to come with a prototype package browser modeled after shopping sites developed in just a few hours?

It's a little Flask webapp that just works on any Debian system, using the local apt-xapian-index as a backend. It has fast keyword search, faceted navigation and screenshots, and it runs on your system showing the packages that you have available.

Screenshot of packageshelf

To try it:

git clone git://git.debian.org/users/enrico/pkgshelf.git
cd pkgshelf
./web-server.py

Then visit http://localhost:5000

It didn't have much interface polishing, as it's just a quick technology demo. However you can see that:

  • keyword search is fast (fast enought that it could be made to search as you type);
  • relevant tags appear on the left, grouped by facets;
  • the most relevant tags are highlighted;
  • the less relevant tags could be hidden behind a [more] expander;
  • you can choose several strategies to hide packages you may find irrelevant.

Things that need doing:

  • hiding uninteresting facets;
  • making it pretty.

It's essentially JavaScript and CSS work. Anyone wants to play?

Posted Sat Jan 22 01:40:50 2011 Tags:

Match package names across distributions

What would happen if we had a quick and reliable way to match package names across distributions?

These ideas came up at the appinstaller2011 meeting:

  • it would be easy to lookup screenshots in the local distro, and if there are none then fall back on other distributions;
  • it would be easy to port Debtags to other distributions, and possibly get changes back;
  • it would be trivial to add a [patches in $DISTRO] link to the PTS
  • it would be easy to point to other BTSes

We thought they were good ideas, so we started hacking.

To try it, you need to get the code and build the index first:

git clone git://git.debian.org/users/enrico/distromatch.git
cd distromatch
# Careful: 90Mb
wget http://people.debian.org/~enrico/dist-info.tar.gz
tar zxf dist-info.tar.gz
# Takes a long time to do the indexing
./distromatch --reindex --verbose

Then you can query it this way:

./distromatch $DISTRO $PKGNAME [$PKGNAME1 ...]

This would give you, for the package $PKGNAME in $DISTRO, the corresponding package names in all other distros for which we have data. If you do not provide package names, it automatically shows output for all packages in $DISTRO.

For example:

$ time ./distromatch debian libdigest-sha1-perl
debian:libdigest-sha1-perl fedora:perl-Digest-SHA1
debian:libdigest-sha1-perl mandriva:perl-Digest-SHA1
debian:libdigest-sha1-perl suse:perl-Digest-SHA1

real    0m0.073s
user    0m0.056s
sys 0m0.016s

Yes it's quick. It builds a Xapian index with the information it needs, and then it reuses it. As soon as I find a moment, I intend to deploy an instance of it on DDE.

It is using a range of different heuristics:

  • match packages by name;
  • match packages by desktop files contained within;
  • match packages by pkg-config metadata files contained within;
  • match packages by [/usr]/bin/* files contained within;
  • match packages by shared library files contained within;
  • match packages by devel library files contained within;
  • match packages by man pages contained within;
  • match stemmed form of development library package names;
  • match stemmed form of shared library package names;
  • match stemmed form of perl library package names;
  • match stemmed form of python library package names.

This list may get obsolete soon as more heuristics get implemented.

Euristics will never cover all corner cases we surely have, but the idea is that if we can match a sizable amout of packages, the rest can be somehow fixed by hand as needed.

The data it requires for a distribution should be rather straightforward to generate:

  1. a file which maps binary package names to source package names
  2. a file with the list of files in all the packages

For example:

$ ls -l dist-debian/
total 39688
-rw-r--r--  1 enrico enrico  1688249 Jan 20 17:37 binsrc
drwxr-xr-x  2 enrico enrico     4096 Jan 21 19:12 db
-rw-r--r--  1 enrico enrico 29960406 Jan 21 10:02 files.gz
-rw-r--r--  1 enrico enrico  8914771 Jan 21 18:39 interesting-files

$ head dist-debian/binsrc 
openoffice.org-dev openoffice.org
ext4-modules-2.6.32-5-4kc-malta-di linux-kernel-di-mipsel-2.6
linux-headers-2.6.30-2-common linux-2.6
libnspr4 nspr
ipfm ipfm
libforks-perl libforks-perl
med-physics debian-med
libntfs-3g-dev ntfs-3g
libguppi16 guppi
selinux selinux

$ zcat dist-debian/files.gz | head
memstat etc/memstat.conf
memstat usr/bin/memstat
memstat usr/share/doc/memstat/changelog.gz
memstat usr/share/doc/memstat/copyright
memstat usr/share/doc/memstat/memstat-tutorial.txt.gz
memstat usr/share/man/man1/memstat.1.gz
libdirectfb-dev usr/bin/directfb-config
libdirectfb-dev usr/bin/directfb-csource
libdirectfb-dev usr/include/directfb-internal/core/clipboard.h
libdirectfb-dev usr/include/directfb-internal/core/colorhash.h

interesting-files and db are generated when indexing.

To prove the usefulness of the idea (but does it need proving?), you can find in the same git repo a little example app (it took me 10 minutes to write it), that uses the distromatch engine to export Debtags tags to other distributions:

$ ./exportdebtags fedora | head
memstat: admin::benchmarking, interface::commandline, role::program, use::monitor
libdirectfb-dev: devel::lang:c, devel::library, implemented-in::c, interface::framebuffer, role::devel-lib
libkonqsidebarplugin4a: implemented-in::c++, role::shared-lib, suite::kde, uitoolkit::qt
libemail-simple-perl: devel::lang:perl, devel::library, implemented-in::perl, role::devel-lib, role::shared-lib, works-with::mail
libpoe-component-pluggable-perl: devel::lang:perl, devel::library, implemented-in::perl, role::shared-lib
manpages-ja: culture::japanese, made-of::man, role::documentation
libhippocanvas-dev: devel::library, qa::low-popcon, role::devel-lib
libexpat-ocaml-dev: devel::lang:ocaml, devel::library, implemented-in::c, implemented-in::ocaml, role::devel-lib, works-with-format::xml
libgnutls-dev: devel::library, role::devel-lib, suite::gnu

Just in case this made you itch to play with Debtags in a non-Debian distribution, I've generated the full datasets for Fedora, Mandriva and OpenSUSE.

Others have been working on the same matching problem. After we started writing code we started to become aware of existing work:

I'd like to make use of those efforts, maybe to cross-validate results, maybe even better as yet another heuristics.

Update:

I built a simple distromatch query system into DDE!

Posted Sat Jan 22 01:40:50 2011 Tags:

fuss-launcher: an application launcher built on apt-xapian-index

Long ago I blogged about using apt-xapian-index to write an application launcher.

Now I just added a couple of new apt-xapian-index plugins that look like they have been made just for that.

In fact, they have indeed been made just for that.

After my blog post in 2008, people from Truelite and the FUSS project took up the challenge and wrote a launcher applet around my example engine.

The prototype has been quite successful in FUSS, and as a consequence I've been asked (and paid) to bring in some improvements.

The result, that I have just uploaded to NEW, is a package called fuss-launcher:

* New upstream release
   - Use newer apt-xapian-index: removed need of local index
   - Dragging a file in the launcher shows the applications that can open it
   - Remembers the applications launched more frequently
   - Allow to set a list of favourite applications

To get it:

  • apt-get install fuss-launcher (after it passed NEW);
  • or git clone http://git.fuss.bz.it/git/launcher.git/ and apt-get install python-gtk2 python-xapian python-xdg apt-xapian-index app-install-data

It requires apt-xapian-index >= 0.35.

To try it:

  1. Make sure your index is up to date, especially if you just installed app-install-data: just run update-apt-xapian-index as root.
  2. Run fuss-launcher.
  3. Click on the new tray icon to open the launcher dialog.
  4. Type some keywords and see the list of matching applications come to life as you type.

It's worth mentioning again that all this work was sponsored by Truelite and the Fuss project, which rocks.

Some screenshots:

When you open the launcher, by default it shows the most frequently started applicationss and the favourite applications:

launcher just opened

When you type some keywords, you get results as you type, and context-sensitive completion:

keyword search

When you drag a file on the launcher you only see the applications that can open that file:

drag files to the launcher

Posted Mon May 17 10:41:09 2010 Tags:
Posted Sat Jun 6 00:57:39 2009

Pages related to my visit in Addis Ababa for a Linux training course.

First day in Addis

First day in Addis Ababa, after the introductory session for this 10 days Linux training.

Interesting new quotes I picked up from the excellent presentation of Dr. Dawit:

Much that I bound I could not free Much that I freed returned to me

(I didn't manage to transcribe the attribution)

And this one for Bubulle, about translation:

When you speak to me in my language you speak to my heart when you speak to me in English you speak to my head

(sb.)

Incomplete list of questions I've been asked, in bogosort -n order:

  • How do I get support?
  • Are the configuration files always the same accross different distributions?
  • What is the level of interoperatibility between the various Linux distributions? And between different Unix-like systems?
  • Does plug and play work well when I change hardware?
  • Can I access NTFS partitions?
  • How do I play multimedia files in restricted formats?
  • I heard that NFS has security problems: can it be secured, or are there other file sharing alternatives?
  • Can I access a desktop remotely?
  • Can I install Linux on a computer where there's Windows already? Do I need to partition?
  • Can I be sure to find drivers for my hardware?

I'm happy to find that we've been successful in building more and more good answers for these questions.

Posted Sat Jun 6 00:57:39 2009 Tags:

First pratical lesson

Notes after today's training session.

Small index of most used shell commands:

  • ls - list directory contents
  • cp - copy files and directories
  • mv - move (rename) files
  • rm - remove files or directories
  • find - search for files in a directory hierarchy
  • cat - concatenate files and print on the standard output
  • more - file perusal filter for crt viewing
  • less - opposite of more (quit with 'q')
  • cd - Change the current directory to DIR. (use "help cd" instead of "man cd")
  • mkdir - make directories
  • rmdir - remove empty directories

Small index of commands useful for combining in pipelines:

  • grep, egrep, fgrep, rgrep - print lines matching a pattern
  • tail - output the last part of files
  • head - output the first part of files
  • sort - sort lines of text files
  • uniq - report or omit repeated lines
  • sed - stream editor
  • wc - print the number of newlines, words, and bytes in files

Problems found during the lesson:

  • You set the system default locale to Amharic, and the gdm login will be in Amharic input mode. We didn't find out how to switch it back to input roman characters. Right click on the input field to set the input method doesn't work. Since usernames are not in Amharic, you're locked out.
  • So you CTRL+ALT+F1, login and try dpkg-reconfigure locales. On Ubuntu Dapper, it does not work anymore.
  • So you dig and dig and dig and finally find that you can force a locale in /etc/default/gdm (but not in /etc/gdm/locale.conf, nor in /etc/gdm/gdm.conf).
  • Then the internet works for a bit and you look up how to reconfigure locales in Ubuntu. Turns out you have to use localeconf, which is not installed by default, is not in universe and thus not on the CDs, and needs to be downloaded from the Internet.
  • The Ubuntu wiki is all on https, which defeats any attempt of proxy caching.
  • An Internet proxy needs to be configured 3 times: in Gnome, in Firefox and in Synaptic (well, apt). This is especially tricky when you forgot to setup the proxy in Synaptic and seemingly unrelated applications fail, like the Ubuntu language selector, which internally invokes the package manager to download missing langpacks.
  • Some short descriptions in the NAME section of manpages are hard to understand, or wrong. Noted on apt-get, apt-cache and less. Top prize goes to apt-cache:

     NAME
            apt-cache - APT package handling utility -- cache manipulator
     DESCRIPTION
            [...] apt-cache does not manipulate the state of the system but
            does provide operations to search and generate interesting output
            from the package metadata. [...]
    

    So apt-cache is a manipulator that doesn't manipulate. A possible improvement can be "query the APT package cache".

  • The language selector in Ubuntu Breezy doesn't really exit and keeps the package database locked. This seems to be fixed in Dapper, and probably had been fixed in some Breezy update. System updates here are a problem: my Dapper (with some Universe things in it) wanted to download more than 120Mb of data, and the Uni network was giving me 14Kbps. It's been a nice opportunity to teach about fuser -uva and kill.
  • dict, squid and many other packages from 'main' are not on the normal Ubuntu CDs: is there an easy way to build a CD with them? Or do Ubuntu CDs with extra packages already exist? I'll have to find out.
  • cupsys has documentation outside of /usr/share/doc, in /usr/share/cups/doc-root.
  • man works on all commands, except cd, which is an internal shell command and thus needs help instead of man. I should remember to ponder about autogenerating manpages from help output.
  • Is there an index-like manpage with a list of the core Unix commands and their short descriptions? It there's not, it's easy to generate:

     #!/bin/sh
     DIR=${1:-"/bin"}
     (
     find $DIR | while read FILE
     do
         if [ -x $FILE ] && ! [ -d $FILE ]
         then
             LANG=C COLUMNS=2000 man `basename $FILE` | \
                      grep ^SYNOPSIS -B 100 | grep ^NAME -A 100 | \
                      tail -n +2 | head -n +2 | \
                      grep -v '^[ \t]*$'
         fi
     done
     ) | sort | uniq | sed 's/^ \+//'
    

    Try running it on /bin and /sbin: it's great!. Also, since it doesn't redirect stderr, it nicely exposes a number of manpage problems.

Lots of bugs to report when I come home: from here it'll take ages, and lots of money on the hotel internet connection, and some are Ubuntu-specific so I'd need to do everything online with Malone.

As usual, teaching is one of the best ways to find bugs.

I propose an Etch training session a month before release.

Other things to do:

  • Find more info about that Wikipedia live CD with Wikipedia browsable without the Internet.
  • Make a collection of Free technical E-books: even those Indian low-cost book editions are too expensive here, so E-books mean a lot.

Update: Matt Zimmerman writes:

I read your blog entry at http://www.enricozini.org/blog/eng/second-day-in-addis and wanted to respond as follows:

  • localeconf is not the standard way to configure locales in Ubuntu; what documentation told you that? It's an unsupported package from Progeny. If what you wanted was to set the system default locale from the command line, editing /etc/environment is probably the best way.

  • I suggest filing a bug report at <https://launchpad.net/products/ubuntu-website about the HTTPS issue>; I don't think it's necessary for the entire wiki to be HTTPS, only authentication.

  • Synaptic may be able to use the GNOME proxy settings without introducing undesirable dependencies; please file a wishlist bug

  • dict, squid and other packages from main are not on the Ubuntu CDs because there is no space. The DVD contains these packages.

  • The cupsys documentation bug was quite likely inherited from Debian and should be reported there

  • You can file bugs in Malone via email; this has been possible for a long time now. Please don't reinforce this misconception.

    https://help.launchpad.net/UsingMaloneEmail

Update:

Posted Sat Jun 6 00:57:39 2009 Tags:

Etiopia

È interessante, bello e triste allo stesso tempo trovarsi a ridefinire il significato di "Abissinia". E maledire che per i primi 30 anni della tua vita, quella parola l'hai sentita soltanto quando uno stronzo cantava "Faccetta nera".

Posted Sat Jun 6 00:57:39 2009 Tags:

Edifici

Da una canzone in amarico:

"Il tuo amore è diventato vecchio

come gli edifici costruiti dagli italiani"

Posted Sat Jun 6 00:57:39 2009 Tags:

Fifth day in Addis

Samba

To get samba:

    apt-get install samba samba-doc smbclient

To get the Samba Web Administration Tool:

    apt-get install swat netkit-inetd

The configuration is in /etc/samba:

  • One [global] section with the general settings
  • One section per share

One could use swat at http://localhost:901/ but it does not work easily on Ubuntu.

To see what is shared:

    smbclient -L localhost

To access a share:

    smbclient //localhost/name-of-the-share

To add a new user:

    sudo smbpasswd -a username

To change the password of a user:

    sudo smbpasswd username

To test accessing a share as a user:

    smbclient //localhost/web -U yared

Documentation:

    man smb.conf

To force the user or group used to access a share:

    force user = enrico
    force group = www-data

To set the unix permissions for every created file:

    # For files
    create mask = 0664
    # For directories
    directory mask = 0775

Example share configuration for a webspace:

    mkdir /var/www/public
    chgrp www-data /var/www/public
    chmod 0775 /var/www/public

Then, in /etc/samba/smb.conf:

    [web]
       comment = Webspace
       path = /var/www
       writable = yes
       public = no
       force group = www-data
       create mask = 0664
       directory mask = 0775

Example share configuration for a read only directory where only a limited group of people can write:

    [documents]
       comment = Documents
       path = /home/enrico/Desktop/documents
       force user = enrico
       public = yes
       writable = no
       write list = enrico, yared

Print server (CUPS)

Installation:

    apt-get install cupsys

Configuration:

  • On the web (not enabled in Ubuntu):

     http://localhost:631/
    
  • On the desktop:

     System/Administration/Printing
    

Example IPP URIs:

    ipp://server[:port]/printers/queue
    http://server:631/printers/queue
    ipp://server[:port]/...

For example:

    ipp://server/printers/laserjet

"This printer uri scheme can be used to contact local or remote print services to address a particular queue on the named host in the uri. The "ipp" uri scheme is specified in the Internet Print Protocol specifications and is actually much more free form that listed above. All Solaris and CUPS based print queues will be accessed using the formats listed above. Access to print queues on other IPP based print servers requires use of the server supported ipp uri format. Generally, it will be one of the formats listed above."

LDAP Lightweight Directory Access Protocol

Installation:

    apt-get install ldap-utils slapd

The configuration is in /etc/ldap.

To access a ldap server:

    apt-get install gq

Various LDAP HOWTOs:

GRUB

The configuration file is in /boot/grub/menu.lst.

The documentation can be accessed as info grub after installing the package grub-doc.

Quick list of keys for info:

  • arrows: move around
  • enter: enters a section
  • l: goes back
  • u: goes up one node
  • q: quit
  • /: search

Grub trick to have a memory checker:

  1. apt-get install memtest86+
  2. Add this to /boot/grub/menu.lst:

    title Memory test
        root (hd0,5)
        kernel /boot/memtest86+.bin
    

Firewall

With iptables:

    man iptables
    # Only allow in input the network packets
    # that are going to the web server
    iptables -P INPUT DROP
    iptables -A INPUT --protocol tcp --destination port 80 -j ACCEPT
    # To reset the input chain as the default
    iptables -F INPUT
    iptables -P INPUT ACCEPT

Some links:

Squid

Installation:

    apt-get install squid

The configuration is in /etc/squid/squid.conf.

To allow the local network to use the proxy:

    # Add this before "http_access deny all"
    acl our_networks src 10.4.15.0/24
    http_access allow our_networks

To use a parent proxy:

    cache_peer proxy.aau.edu.et     parent    8080  0  proxy-only no-query

Pay attention because /var/spool/squid will grow as the cache is used. The maximum cache size is set in the directive cache_dir.

Information about squid access control is at http://www.squid-cache.org/Doc/FAQ/FAQ-10.html

To check that the configuration has no syntactic errors: squid -k parse.

To match urls:

    acl forbiddensites url_regex [-i] regexp

For info about regular expressions:

    man regex

Example filtering by regular expression:

    acl skype url_regex -i [^A-Za-z]skype[^A-Za-z]
    http_access deny skype

Transparent proxy setup: http://www.tldp.org/HOWTO/TransparentProxy.html

Problems found today

Hiccups of the day:

  • swat does not run on Ubuntu because Ubuntu does not have inetd
  • swat does not allow root login on Ubuntu because root does not have a password
  • smbpasswd -a does not seem to update the timestamp of /var/lib/samba/passwd.tdb
  • cups web admin does not work on Ubuntu
  • LDAP is still not so intuitive to set up

Update: Marius Gedminas writes:

I think it would be a good idea to mention that running

     iptables -P INPUT DROP

in the shell is a Bad Idea if you're logged in remotely via SSH.

Posted Sat Jun 6 00:57:39 2009 Tags:

Fourth day in Addis

Unix file permissions:

    drwxr-xr-x   2 root root    38 2006-07-14 
    |
    +- Is a directory

    drwxr-xr-x   2 root root    38 2006-07-14 
     ---
      |
      +- User permissions (u)

    drwxr-xr-x   2 root root    38 2006-07-14 
        ---
         |
         +- Group permissions (g)

    drwxr-xr-x   2 root root    38 2006-07-14 
           ---
            |
            +- Permissions for others (o)

    drwxr-xr-x   2 root root    38 2006-07-14 
                   ----
                    |
                    +- Owner user

    drwxr-xr-x   2 root root    38 2006-07-14 
                        ----
                         |
            Owner group -+

Other bits:

  • 4000 Set user ID:

    • For executable files: run as the user who owns the file, instead of the user who runs the file
    • For directories: I think it's not used
  • 2000 Set group ID:

    • For executable files: run as the group who owns the file, instead of the group of the user who runs the file
    • For directories: when a file is created inside the directory, it belongs to the group of the directory instead of the default group of the user who created the file
  • 1000 Sticky bit:

    • For files: I think it's not used anymore
    • For directories: only the owner of a file can delete or rename the file

The executable bit for directories means "can access the files in the directory".

If a directory is readable but not executable, then I can see the list of files (with ls) but I cannot access the files.

To access a file, all the directories of its path up to / need to be executable.

Commands to manipulate permissions:

  • chown - change file owner and group
  • chgrp - change group ownership
  • chmod - change file access permissions

  • sudo adduser enrico www-data adds the user enrico to the group www-data.

Example setup for a website for students:

    # Create the group 'students'
    mkdir /var/www/students
    chgrp students /var/www/students
    chmod 2775 /var/www/students

    # If you don't want other users to read the files of the students:

    chmod 2770 /var/www/students
    adduser www-data students
     (this way the web server can read the
      pages)

    # when you add a user to a group, it does not affect running processes:

     - users need to log out and in again
     - servers need to be restarted

Apache:

  • To install apache2 without a graphical interface:

     apt-cache search apache2 | less
     sudo apt-get install apache2
    
  • By default, /var/www is where is the static website.

  • By default, ~/public_html is the personal webspace for every user, accessible as: http://localhost/~user

  • By default, /usr/lib/cgi-bin contains scripts that are executed when someone browses http://website/cgi-bin/script

  • By default, apache reads the server name from the DNS. If we don't have a name in the DNS and we want to use the IP, we need to set:

     ServerName 10.4.15.158
    

    in /etc/apache/apache2.conf (set it to your IP address)

  • To access the Apache manual: http://localhost/doc/apache2-doc/manual/

  • http://localhost/doc/apache2-doc/manual/mod/mod_access.html The access control module

  • http://localhost/doc/apache2-doc/manual/mod/mod_auth.html The user authentication module

  • To edit a user password file, use:

     htpasswd - Manage user files for basic authentication
    
  • Example .htaccess file to password protect a directory:

     AuthUserFile /etc/apache2/students
     AuthType Basic
     AuthName "Students"
     Require valid-user
    
  • Information about .htaccess is in http://localhost/doc/apache2-doc/manual/howto/htaccess.html

  • If you need to tell apache to listen on different ports, add a Listen directive to /etc/apache2/ports.conf. Then you can use:

     <VirtualHost www.training.aau.edu.et:9000>
     [...]
     </VirtualHost>
    
  • To setup an HTTPS website:

    • Documentation is in http://localhost/doc/apache2-doc/manual/ssl/
    • How to create a certificate: http://www.tc.umn.edu/~brams006/selfsign.html

    • Create a certificate:

      /usr/sbin/apache2-ssl-certificate -days 365

    • Create a virtual host on port 443:

      [...]

    • Enable SSL in the VirtualHost:

      SSLEngine On SSLCertificateFile /etc/apache2/ssl/apache.pem

    • Enable listening on the HTTPS port (/etc/apache2/ports.conf):

      Listen 443

Apache troubleshooting:

  • check that there are no errors in the configuration file:

     apache2ctl configtest
    

    This it is always a good thing to do before restarting or reloading apache.

  • read logs in /var/log/apache2/

  • if you made a change but you don't see it on the web, it can be that you have the old page in the cache of the browser: try reloading a few times.

To install PHP

  • apt-get install libapache2-module-php5
  • then by default, every file .php is executed as php code
  • Small but useful test php file:

     <? phpinfo() ?>
    

To install MySQL

  • apt-get install mysql-client mysql-server
  • for administration run mysql as root:

    • Create a database with:

      create database students

  • Give a user access to the database:

     # Without password
     grant all on students.* to enrico;
    
     # With password
     grant all on students.* to enrico identified by "SECRET";
    
  • More information can be found at http://www-css.fnal.gov/dsg/external/freeware/mysqlAdmin.html

To use MySQL from PHP:

    apt-get install php5-mysqli php5-mysql

Problems found today:

  • the apache2 manual in /usr/share/doc/manual can only be viewed using apache because it uses MultiView. So you need to have a working apache to read how to have a working apache.

  • chmod does not have examples in the manpage.

Posted Sat Jun 6 00:57:39 2009 Tags:

Eight day in Addis

Useful things to keep in mind when setting up a service:

  • always take note of what you do
  • make yourself always able to explain to another person what you did
  • keep a copy of the configuration files before changing them, so that you can see what you changed
  • be always able to move the service to another computer
  • make sure that it works after reboot

Example use of vim block selection:

  • ESC: exits insert mode.
  • ^V: starts block selection. Move the arrows to form a rectangle.
  • c: change. Type the new content for the line.
  • ESC: gets out of insert mode, and the change will happen in all the lines.

To change network configuration with config files, edit:

/etc/network/interfaces

To also setup DNS in /etc/network/interfaces, use dns-search and dns-nameservers (for this to work, you need to have the package resolvconf):

dns-search dream.edu.et
dns-nameservers 192.168.0.1 192.168.0.2

To make a router that connects to the internet on demand using a modem:

apt-get install diald

To see the path of network packets:

mtr 4.2.2.2

Basic NAT script:

OUT=eth2
IN=eth0

modprobe iptable_nat
iptables -t nat -A POSTROUTING -o $OUT -j MASQUERADE
echo 1 > /proc/sys/net/ipv4/ip_forward

What happens at system startup:

  1. the BIOS loads and runs the boot loader
  2. the boot loader loads the kernel and the inintrd ramdisk and runs the kernel
  3. the kernel runs the script 'init' in the initrd ramdisk
  4. the script 'init' mounts the root directory
  5. the script 'init' runs the command /sbin/init in the new root directory
  6. 'init' starts the system with the configuration in /etc/inittab

To install a new startup script:

sudo ln -s /usr/local/sbin/firewall /etc/init.d
sudo update-rc.d firewall defaults 16 75

Normally you can just do:

sudo update-rc.d [servicename] defaults

To have a look at the start and stop order numbers, look at /etc/rc2.d for other start scripts and /etc/rc0.d for other stop scripts

To test a proxy, low level way:

$ telnet proxy 8080
Trying 192.168.0.6...
Connected to proxy.dream.edu.et.
Escape character is '^]'.
GET http://www.google.com HTTP/1.0 [press enter twice]
Posted Sat Jun 6 00:57:39 2009 Tags:

Ninth day in Addis

SSH

To enable remote logins with ssh

apt-get install openssh-server

Then you can login with:

$ ssh efossnet@proxy.dream.edu.et

To verify the host key fingerprint of a machine:

$ ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub

Note: you need to verify it before logging in!

More information at http://www.securityfocus.com/infocus/1806

Example ssh usages

To log in:

    $ ssh efossnet@proxy

To run a command in the remote computer:

    $ ssh efossnet@proxy "cat /etc/hosts"

To copy a file to the remote computer:

    $ scp Desktop/july-18.tar.gz efossnet@proxy:

To copy a file from the remote computer:

    $ scp efossnet@proxy:july-18.tar.gz /tmp/

Beware of brute-force login attempts

Warning about SSH: there are people who run automated scans for ssh servers and try to login using commonly used easy passwords.

If you have an SSH server on the network, use strong passwords, or if you can it's even better to disable password authentication: in /etc/ssh/sshd_config, add:

    PasswordAuthentication no

To log in using public/private keys:

  1. Create your key:

    ssh-keygen -t rsa
    
  2. Copy your public key to the machine where you want to log in:

    ssh-copy-id -i .ssh/id_rsa.pub efossnet@proxy
    
  3. Now you can ssh using your RSA key

If you use ssh often, read these:

proxy

Problems we had today with the proxy:

ssl does not work

Reason: squid tries to directly connect to the ssl server, but the AAU network wants us to go through their proxy.

Ideal solution: none. There is no way to tell squid to use a parent proxy for SSL connections.

Solution: update the documentation for the Dream university users telling to setup a different proxy for SSL connections.

Longer term solution: get the AAU network admins to enable outgoing SSL connections from the Dream university proxy.

Other things that can be done:

  • report a bug on squid reporting the need and requesting the feature
  • download squid source code and implement the feature ourselves, then submit the patch to the squid people

Browsing normal pages returns an error of 'Connection refused'.

In the logs, the line is:

1153294204.912    887 192.168.0.200 TCP_MISS/503 1441 GET http://www.google.com.et/search? - NONE/- text/html

That "/503" is one of the HTTP error codes.

Explanation of the error codes:

Reason: the other proxy is refusing connections from our proxy.

Solution: none so far. Will need to get in touch with the admins of the other proxy to try to find out why it refuses connection to our proxy, and how we can fix the problem.

postfix on smtp.dream.edu.et

Basic information is at http://www.postfix.org/basic.html.

Difference between mail name and smarthost:

  • The mail name is the name of the mail server you're setting up (TODO: need more details on what's it used for)
  • The smarthost is the name of the mail server that will relay mail for you.

Quick way to send test mails:

apt-get install mailx
echo ciao | mail efossnet@localhost

To configure a workstation not to do any mail delivery locally and send all mail produced locally to smtp.dream.edu.et:

  1. install postfix choosing "Satellite system"
  2. put smtp.dream.edu.et as a smarthost.

To setup a webmail: apt-get install squirrelmail (on a working apache setup).

To setup mailing lists: apt-get install mailman, then follow the instructions in /usr/share/doc.

Mail server issues we encountered

When a mail is sent to efossnet@localhost, the system tries to send it to efossnet@yoseph.org

Investigation:

  • "yoseph.org" does not appear anywhere in /etc or /var/spool/postfix
  • postfix configuration has been reloaded
  • postfix logs show that the mail has been 'forwarded'

Cause: the user efossnet had forgotten that he or she had setup a .forward file in the home directory.

Solution:

 rm ~efossnet/.forward

Apache

To add a new website:

  1. cd /etc/apache2/sites-available
  2. sudo cp default course
  3. sudo vi course:

    1. Remove the first line
    2. Add a ServerName directive with the address of your server: ServerName course.dream.edu.et
    3. Customize the rest as needed: you at least want to remove the support for browsing /usr/share/doc and you want to use a different document root.
  4. sudo a2ensite course

  5. sudo /etc/init.d/apache2 reload

More VIM

Undo: u (in command mode)

Redo: ^R (in command mode)

You can undo and redo multiple times.

To recover a lost password for root or for the ubuntu admin user

Boot with a live CD, mount the system on the hard disk (the live CD usually does it automatically), then edit the file /etc/shadow, removing the password:

enrico:$1$3AJfasjJFHa234dfh230:13343:0:99999:7:::

becomes:

enrico::13343:0:99999:7:::

You can edit the file because, in the live CD system, you can always become root.

After you do this, reboot the system: you can log in without password, and set yourself a new password using the command passwd.

Installing packages not on the CDs

To get a package for installing when offline:

  1. apt-get --print-uris install dnsmasq
  2. Manually download the packages at the URLs that it gives you

Otherwise, apt-get --download-only install dnsmasq will download the package for you in /var/cache/apt/archives.

You can install various previously downloaded debian packages with:

dpkg -i *.deb

Backups

There are various ways:

  • dump (for ext2/ext3 file systems) or xfsdump (for xfs file systems).

    Makes a low-level dump of the file system.

    It must be used for every different partition.

    It makes the most exact backup possible, including inode numbers.

    It can do full and incremental backups.

    To see the type of the filesystems, use 'mount' with no parameters.

    To restore: restore or xfsrestore.

  • tar

    Filesystem independent.

    It can work accross partitions.

    It correctly backups permissions and hard links.

    It can do full and incremental backups.

    Example:

     tar lzcpf backup.tar.gz /home /var /etc /usr/local
     tar lzcpf root.tar.gz /
    

    To restore:

     tar zxpf backup.tar.gz
    
  • faubackup

    Filesystem independent.

    Uses hard drive as backup storage.

    Always incremental.

    It cannot do compression.

    Unchanged files in new backups are just links to old backups, and do not occupy space.

    Any old backup can be deleted at any time without compromising the others.

    It can be used to provided a "yesterday's files" service to users (both locally and exported as a read-only samba share...).

    To restore, just copy the files from the backup area.

  • amanda

     apt-get install amanda-client amanda-server
    

    It is a network backup system.

    It can do full and incremental backups.

    You can have a backup server which handles the storage and various backup clients that send the files to backup to the server.

    It takes some studying to set up.

    To restore: it has its own tool.

Some data requires exporting before backing it up:

  • To save the list of installed packages and the answer to configuration questions:

     dpkg --get-selections > pkglist
     debconf-get-selections > pkgconfig
    

    To restore:

     dpkg --set-selections < list
     debconf-set-selections < pkgconfig
     apt-get dselect-upgrade
    

    If you do this, they you only need to backup /etc, /home, /usr/local, /var.

  • To save the contents of a MySQL database:

     mysqldump name-of-database | gzip > name-of-database.dump.gz
    

    To restore:

     zcat name-of-database.dump.gz | mysql
    

You can schedule these dumps to be made one hour before the time you make backups.

Scheduling tasks

As a user:

crontab -e

As root: add a file in one of the /etc/cron.* directories.

In cron.{hourly,daily,weekly,monthly} you put scripts.

In the other directories you put crontab files (man 5 crontab).

If the system is turned off during normal maintainance hours, you can do two things:

  1. Change /etc/crontab to use different maintanance hours
  2. Install anacron (it's installed by default in ubuntu)

For scheduling one-shot tasks, use at(1):

$ at 17:40
echo "Please tell Enrico that the lesson is finished" | mail efossnet@dream.edu.et
^D

When and how to automate

  1. First, you manage to do it yourself
  2. Then, you document it
  3. Then, you automate it

Start at step 1 and go to 2 or 3 if/when you actually need it.

(credits to sto@debian.org: he's the one from which I heard it for the first time, said so well).

Interesting programs to schedule during maintanance

  • rkhunter, chkrootkit
  • checksecurity
  • debsecan
  • tiger

Important keys to know in a Unix terminal

These are special keys that work on Unix terminals:

  • ^C: interrupt (sends SIGTERM)
  • ^\: interrupt (send SIGQUIT)
  • ^D: end of input
  • ^S: stop scrolling
  • ^Q: resume scrolling

Therefore, if the terminal looks like it got stuck, try hitting ^Q.

Problems we had today with postfix

  • Problem: mail to efossnet@dream.edu.et is accepted only if sent locally.

    Reason:

     $ host -t mx dream.edu.et
     Host dream.edu.et not found: 3(NXDOMAIN)
    

    Solution: tell dnsmasq to handle a MX record also for dream.edu.et:

    mx-host=dream.edu.et,smtp.dream.edu.et,50

  • The problem not solved with the previous solution.

    Reason: postfix was making complaints which mentioned localhost as a domain name.

    Solution: fixed by changing 'myhostname' in main.cf to something different than localhost.

    Note: solved by luck. Investigate why this happened.

Problems found yesterday and today

  • there is no way to tell squid to use another proxy for SSL connections: it only does them directly
  • if you want to configure evolution to get mail from /var/mail/user, you need to explicitly enter the path. It would be trivially easier if evolution presented a good default, since it's easy to compute. It would also be useful if below the "Path" entry there were some text telling what path is being requested: the mail spool? the evolution mail storage?
  • In Evolution: IMAP or IMAPv4r1? What is the difference? Why should I care?
  • apt-get --print-uris doesn't print the URIs if the package is in the local cache, and there seems to be no way to have it do it.
  • in /etc/apache2/sites-available/default, is the NameVirtualHost * directive appropriate there? It gets in the way when using 'default' as a template for new sites.

    Otherwise, one can add a new (disabled) site that can be used as a template for new sites instead of default.

  • the default comments put by crontab -e are not that easy to read.

Posted Sat Jun 6 00:57:39 2009 Tags:

Addis course Tasks & Skills questions

  • What does the command find /etc | less do?

  • What does the command ps aux do?

  • What does the command mii-tool do and when would you use it?

  • What does the command host www.google.com do?

  • How do you get the MAC address of your computer?

  • What can you use dnsmasq for?

  • What is in /etc/dnsmasq.conf?

  • What is the use of the dhcp-option configuration parameter of /etc/dnsmasq.conf?

  • What is the difference between chown, chgrp and chmod?

  • What would you use nmap for?

  • How do you check to see if a network service is running on your computer?

  • What does apache2ctl configtest do? When should you run it?

  • Consider this piece of configuration of apache:

     AuthUserFile /etc/apache2/students
     AuthType Basic
     AuthName "Students"
     Require valid-user
    

    What does it do?

    What command would you use to add a new username and password to /etc/apache2/students? (you can write the entire commandline if you know it, but just the name of the command is fine)

  • You created the configuration for a new apache site in /etc/apache2/sites-available. How do you activate the new site?

  • When do you need to add the line Listen 443 to /etc/apache2/ports.conf?

  • What do you normally find in /var/log/syslog, and when would you read it?

  • What does the command smbclient //localhost/web do?

  • What does the command sudo smbpasswd -a enrico do?

  • Where do you look for the explanation of the many directives found in /etc/samba/smb.conf?

  • What is the purpose of the package cupsys?

  • What is the purpose of the command iptables?

  • What is the difference between MDA, MTA and MUA?

  • In a normal mail server configuration, when should you accept a mail coming from outside your local network?

  • Suppose you are a mail software and you need to send a mail to addis@yahoo.com: how do you find out the internet host to which you should connect to send the mail?

  • What is the difference between man 5 postconf and man 8 postconf?

  • What is the different use of SMTP and IMAP?

  • What is a "smarthost" in the context of mail server configuration?

  • What does the command mailq do?

  • What does the command sudo postsuper -d ALL deferred do?

  • Postfix has four mail queues: "incoming", "active", "deferred" and "hold". What is the difference among them?

  • What does the package dovecot do?

  • In the file /etc/dovecot/dovecot.conf, what is the difference between having protocols = imap and protocols = imaps?

  • What happens if I put the line enrico@enricozini.org in the file /home/enrico/.forward?

  • Consider this list of possible strategies for handling mail classified as spam:

    • silently delete it
    • refuse the mail and send a notification to the sender
    • refuse the mail and send a notification to the receiver
    • quarantine the e-mail
    • refuse delivery with a SMTP error
    • deliver with an extra header that says that it's spam

    What are their advantages and disadvantages?

Posted Sat Jun 6 00:57:39 2009 Tags:

Seventh day in Addis

Setting up a mail server

Background

Some terminology:

  • MTA: Mail Transport Agent
  • MUA: Mail User Agent
  • MDA: Mail Delivery Agent
  • SMTP: Simple Mail Transfer Protocol
  • MX: Mail eXchange
  • POP: Post Office Protocol
  • IMAP: Internet Message Access Protocol

With SMTP you connect to a server and send two things: envelope and message.

The envelope looks like this:

MAIL FROM: <enrico@enricozini.org>
RCPT TO: <rms@fsf.org>
RCPT TO: <linus@linux.org>

The message looks like this:

From: <enrico@enricozini.org>
To: <rms@fsf.org>
Cc: <linus@linux.org>
Message-ID: <1234567@enricozini.org>
Subject: Test mail

Hi Richard,

this is a test mail.  I'm also writing
Linus to show how to send to more people.

Cheers,

Enrico

There is no authentication.

There is no encryption.

Two usual types of access control:

  1. Outbound e-mail is normally only accepted from an internal network
  2. Inbound e-mail is normally accepted from anywhere

The DNS is used to find the SMTP server to use to send a message:

$ host -t MX yahoo.com
yahoo.com MX 10 smtp1.yahoo.com
yahoo.com MX 20 smtp2.yahoo.com
yahoo.com MX 20 smtp3.yahoo.com

The process of sending an E-Mail:

  1. Enrico writes an E-Mail:

    From: Enrico Zini <enrico@enricozini.org>
    To: Richard Stallman <rms@fsf.org>
    Subject: Hello from Addis
    
    Hi Richard,
    
    Addis is a wonderful city, even if
    it rains a lot.
    
    Bye,  Enrico
    
  2. Enrico's MUA connects to the SMTP server (for example, port 25 of smtp.aau.edu.et):

    HELO enricozini.org
    200 OK Hello enricozini.org
    MAIL FROM: <enrico@enricozini.org>
    200 OK Mail from enrico@enricozini.org
    RCPT TO: <rms@fsf.org>
    

    Here, the SMTP server performs relay control: "do we relay mail to rms@fsf.org?":

    • Outbound e-mail is normally only accepted from an internal network
    • Inbound e-mail is normally accepted from anywhere

    A target address could be refused:

    413 ERR I don't relay for rms@fsf.org
    

    In this case, the destination is not local but the recipient is accepted because I'm inside the local network:

    200 OK Destination rms@fsf.org
    DATA
    200 OK Please send message body
    From: Enrico Zini <enrico@enricozini.org>
    To: Richard Stallman <rms@fsf.org>
    Subject: Hello from Addis
    Date: Mon, 17 Jul 2006 09:49:45 +0300
    Message-ID: <124372643@enricozini.org>
    
    Hi Richard,
    
    Addis is a wonderful city, even if
    it rains a lot.
    
    Bye,  Enrico
    .
    200 OK Message accepted
    QUIT
    200 OK Bye.
    
  3. The SMTP server needs to find out where to send the message, using the DNS:

    $ host -t MX fsf.org
    fsf.org MX 10 mail.fsf.org
    fsf.org MX 20 mail.gnu.org
    
  4. So the SMTP server tries the first one and connects to port 25 of mail.fsf.org:

    HELO smtp.aau.edu.et
    200 OK Hello smtp.aau.edu.et
    MAIL FROM: <enrico@enricozini.org>
    200 OK Mail from enrico@enricozini.org
    RCPT TO: <rms@fsf.org>
    

    The destination is accepted because it's for a local user::

    200 OK Destination rms@fsf.org
    DATA
    200 OK Please send message body
    From: Enrico Zini <enrico@enricozini.org>
    To: Richard Stallman <rms@fsf.org>
    Subject: Hello from Addis
    Date: Mon, 17 Jul 2006 09:49:45 +0300
    Message-ID: <124372643@enricozini.org>
    Received: by mail.aau.edu.et
      on Mon, 17 Jul 2006 09:55:53 +0300
      from 10.4.15.158
    
    Hi Richard,
    
    Addis is a wonderful city, even if
    it rains a lot.
    
    Bye,  Enrico
    .
    200 OK Message accepted
    QUIT
    200 OK Bye.
    
  5. Now, mail.fsf.org will invoke a MDA to write the mail in Richard Stallman's mailbox.

Example of problems with mail handling:

  • Accepting inbound connections:
    • Malicious input:
      • logic errors
      • buffer overflows
      • DoS (Denial Of Service) attacks
      • Connection floods
  • Performing outbound connections:
    • Programming errors:
      • Flooding of connections
  • Performing routing:
    • Unauthorised relays
    • Mail loops
  • Writing to the local hard drive:
    • Filling up the hard drive
    • Writing to the wrong files
  • Writing to the local hard drive as root:
    • In case of error or attack, any file in the system can potentially be compromised

RFC-822 is the original standard for E-mail. RFCs are standard Internet documents. Have a look at RFC documents released the 1st of April.

postfix

Common setup: "Internet site with smarthost".

More difficult to maintain: "Internet site".

A smarthost is a machine that will relay e-mail for you.

Questions asked with "Internet site with smarthost":

  • Mail name: aau.edu.et (name used to publicly identify the mail server)
  • Smarthost name: smtp.telecom.net.et (SMTP server that will relay our e-mail)

To test a mail server::

$ telnet localhost 25
HELO me
MAIL FROM: <a@b.c>
RCPT TO: <mail@of.a.local.user>
DATA

hi
.
QUIT

By default, you find locally delivered mail in /var/mail/username.

Postfix configuration files:

  • /etc/postfix/master.cf: configures how all the postfix components run together (man 5 master)
  • /etc/postfix/main.cf: Main postfix configuration (man 5 postconf)

To rewrite addresses:

  1. In /etc/postfix/main.cf::

    canonical_maps = hash:/etc/postfix/canonical
    
  2. Then in /etc/postfix/canonical you can add the rewrite rules, like::

    enrico   enrico@enricozini.org
    
  3. When /etc/postfix/canonical is modified you need to regenerate the index::

    sudo postmap canonical
    

    (same is when you change the alias file: sudo postalias /etc/aliases)

(see file:///usr/share/doc/postfix/html/ADDRESS_REWRITING_README.html)

Manipulating the message queue:

mailq - List the mail queue.

Example::

    mailq

postqueue - Postfix queue control

Examples::

    # Like mailq
    postqueue -p

    # Tries to send every message in the queue
    postqueue -f

    # Tries to send every message in the queue for that site
    postqueue -s site

postsuper - Postfix superintendent

Examples::

    # Deletes one message
    sudo postsuper -d 7C4D2EC0F5D

    # Deletes all messages held in the queue for later delivery
    sudo postsuper -d ALL deferred

Different mail queues in postfix:

  • incoming: mail who just entered the system
  • active: mail to be delivered
  • deferred: mail to be delivered later because there were problems
  • hold: mail that should not be delivered until released from hold

Mail logs are in::

/var/log/mail.log
/var/log/mail.err
/var/log/mail.info
/var/log/mail.warn

Mail delivery

Mailbox formats:

  • mbox: single file, mail separated by "From " lines
  • maildir: one directory per folder, one file per mail
  • mh: similar to maildir, but not really used

Alternate MDA: procmail: allows to filter mail automatically into different folders.

Mail forwarding with ~/.forward: allows to redirect mail to a different address: just put the address you want to send to in the file ~/.forward.

POP or IMAP server

Installation:

apt-get install dovecot

Configuration is in::

/etc/dovecot/dovecot.conf

The main thing that is needed is to enable the mail protocols you want::

protocols = imaps

Server monitoring

To make all sorts of graphs::

apt-get install munin munin-node

Example: http://munin.ping.uio.no

To compute more statistics:

  • anteater
  • isoqlog
  • mailgraph

Monitor system logs: logcheck:

  • sends you mail with abnormal log lines
  • It's important to customize what is normal and you do it with regular expressions

Filtering viruses and spam

clamav - Virus scanner

Virus scanning:

  • Postfix gives the mail to clamav that scans it and gives it back if it's clean.
  • Strategies for infected mail:
    • silently delete it
    • refuse the mail and send a notification to the sender
    • refuse the mail and send a notification to the receiver
    • quarantine the e-mail
    • refuse delivery with a SMTP error
    • deliver with an extra header that says that it's a virus

spamassassin - Spam filter

Spam scanning:

  • Postfix gives the mail to spamd that scans it and gives it back with some spam information.
  • Strategies for spam mail:
    • silently delete it
    • refuse the mail and send a notification to the sender
    • refuse the mail and send a notification to the receiver
    • quarantine the e-mail
    • refuse delivery with a SMTP error
    • deliver with an extra header that says that it's spam
  • New techniques:
    • greylisting: when you receive a mail from a host you've never seen before, refuse it with a temporary error, and accept it the second time (after some time delay). Spammers normally don't retry, and implementing retry would increase their cost of sending e-mail.
    • crossassassin: if more than some amount of your users receive a mail with the same message ID, throw it away. Sending mails with different headers would increase the cost of sending e-mail.

Man pages and sections

Man pages are divided in sections:

  • man man shows all the sections of the manpages
  • man 5 postconf shows the postconf manpage in the "configuration file" section
  • Normally manpages are referred as manpage(section) (e.g. postconf(5) )

Authentication and encryption with SMTP (update by Marius Gedminas)

You can have authentication and encryption with SMTP:

Cheat sheet

Setting up the client (I assume Ubuntu)

  # vi /etc/postfix/main.cf

      relayhost = [hostname.of.your.ISPs.smtp.server]
      smtp_use_tls = yes
      smtp_enforce_tls = yes
      smtp_tls_enforce_peername = no
      smtp_sasl_auth_enable = yes
      smtp_sasl_password_maps = hash:/etc/postfix/smtp_auth
      smtp_sasl_security_options = noanonymous

  # vi /etc/postfix/smtp_auth

      [hostname.of.your.ISPs.smtp.server] username:password

  # chmod 600 /etc/postfix/smtp_auth
  # postmap /etc/postfix/smtp_auth
  # postfix reload

(It would be a good idea to make the client verify the server's certificate to prevent man-in-the-middle attacks, but I haven't figured out that part yet...)

Setting up the server

  # apt-get install sasl2-bin libsasl2-modules
  # saslpasswd2 -u hostname.of.the.server -c username1
  # saslpasswd2 -u hostname.of.the.server -c username2
  ...

        these commands create /etc/sasldb2

  # echo "pwcheck_method: auxprop" > /etc/postfix/sasl/smtpd.conf
  # touch /var/spool/postfix/etc/sasldb2
  # echo mount --bind /etc/sasldb2 /var/spool/postfix/etc/sasldb2 \
          > /etc/init.d/local-sasl-for-postfix
  # chmod +x /etc/init.d/local-sasl-for-postfix
  # ln -s ../init.d/local-sasl-for-postfix /etc/rc2.d/S19local-sasl-for-postfix
  # /etc/init.d/local-sasl-for-postfix
  # adduser postfix sasl

        these commands let postfix (which runs chrooted) access /etc/salsdb2

  # cd /etc/postfix
  # openssl req -new -outform PEM -out smtpd.cert -newkey rsa:2048 -nodes \
            -keyout smtpd.key -keyform PEM -days 365 -x509
  # chmod 600 smtpd.key

        these commands create a self-signed SSL certificate

  # vi main.cf

      smtpd_sasl_auth_enable = yes
      broken_sasl_auth_clients = yes
      smtpd_sasl_local_domain = hostname.of.the.server
      smtpd_recipient_restrictions = permit_mynetworks,
                                     permit_sasl_authenticated,
                                     reject_unauth_destination
      smtpd_use_tls = yes
      smtpd_tls_cert_file = /etc/postfix/smtpd.cert
      smtpd_tls_key_file = /etc/postfix/smtpd.key

  # /etc/init.d/postfix restart
Posted Sat Jun 6 00:57:39 2009 Tags:
Posted Sat Jun 6 00:57:39 2009
dcg

Debian Community Guidelines.

DCG mentioned on Linux.com

A friend of mine pointed me to the Debian Community Guidelines being mentioned in a linux.com article.

"[Mako] hopes that Garrett's resignation will give the Debian community an added impetus to adapt its own code of conduct, like the one proposed by Enrico Zini."

I'm very happy to see the DCG geting mentioned, although I don't think that it makes sense to 'adopt' such a document.

what I'd like for it is to be mentioned as a suggested read, and linked from here and there. So today it happened, and I'm happy :)

Posted Sat Jun 6 00:57:39 2009 Tags:

Debconf6 talks material now online

I've finally put online slides and notes for my debconf6 talks:

Many people had asked me the notes for the "Advanced ways of wasting time" talk: they're finally online, translated and with the links pointing to English Wikipedia pages. Sorry it took me so long.

Posted Sat Jun 6 00:57:39 2009 Tags:

Reorganization of the DCG

I've recently received a substantial amount of feedback about the Debian Community Guidelines and went into some reorganization of it.

The previous general section still stands as the Main Guidelines: those are the substantial few things to always keep in mind.

What previously was the long list of checklists is now split in two: the Debian-specific Guidelines, which should be a shorter lists of non-obvious suggestions for people who already have experience with online life, and the General Guidelines, with the fuller checklists with useful suggestions for everyone.

I still haven't gone through the selection and reorganization of the Debian-specific and General part, so at the moment they look fairly similar and most things overlap. But the good news is that I finally found a structure that I like, and that can allow more experienced people to make use of the guidelines without getting bored with simpler things like "google before asking a question".

This division also suggests a little workflow: new suggestions can be added to the Debian-specific part, and then later moved to the general part when they become obvious for everyone.

I'm happy. This layout seems to be good in getting me unstuck with how to think of the DCG. More will come of course, as I'll prepare my DCG talk for Debconf6.

Posted Sat Jun 6 00:57:39 2009 Tags:

Converging to a solution

Sustain a discussion towards solving a problem is sometimes more important than solving the problem.

I can't decide if this is trivial or counterintuitive. Anyway it's been quite enlightening when it came out. I once took this note:

I found that with my projects, when someone posted a mail about a problem I would work maybe some days to find a solution, and just post the solution at the end.

However, now I realised it's more costructive to have the problem-solving process itself happen online. This way, instead of keeping people waiting in silence for a few days they can get quicker feedback and extra informations, and they also have a chance to participate to solving the problem before I manage to.

For example, when I have to interrupt to go home or sleep, someone else can pick it up and do another step.

Plus, the entire problem-solving process remains documented, which will provide more written information for future readers.

This note was from a few months ago; however, I still fail to do it. Bad habits are sometimes hard to change. Please kick me about it :)

Posted Sat Jun 6 00:57:39 2009 Tags:
Posted Sat Jun 6 00:57:39 2009

Pages about Ubuntu.

Live CD on a removable disk

Eros is a hardware guru that happened to be the unknown guy sitting next to me on a plane.

He happens to be a happy Kubuntuer. While chatting, he told me one of his systems is an external hard drive made by copying a Kubuntu live CD image on it.

Why did you do so? I asked.

Because this way I can plug it in any computer, and it'll do hardware detection at boot. However it's a hard drive, so it's fast, and I can keep my home and all my customisations on it.

I had never thought of it.

That's an interesting and smart (ab)use of a live CD.

Now I wonder: what would be required to plug the live CD boot time hardware detection infrastructure on an existing Debian or Ubuntu instalation?

Update: slh on IRC suggests (a bit edited by me):

A lot of the former "obscure black magic" for live CDs isn't needed anymore. What is needed is: a kernel with static usb-storage, libusual, ehci-hcd, ohci-hcd, uhci-hdc (or an appropriate initrd/ initramfs). udev takes care of most h/w detection issues these days.

As long as everything needed to boot is contained in a single partition you don't need a fstab: udev, hal and pmount take care of the rest, procfs, sysvfs, devpts, usbfs, shm are mounted by sysvinit.

All what is left is a tool to create the xorg.conf while booting (those tools exist and just need to be called early).

Everything else is just a matter of convenience: enhancing the live span of the USB key by changing data into tmpfs, etc.; if passwordless logins are required then xsession and inittab need to be changed; new ssh host keys generated on boot; small stuff.

With ordinary flash storage, jffs2 and something to reduce write access is a good idea: perhaps unionfs for /var/ and /home/, bind mounting /tmp/ on /var/tmp/), but that's also not strictly necessary.

Mostly it boils down to running the xorg-creation script at every boot time.

There are various tools to do that. Some are here, but there is surely more. (Enrico's note: do we have anything in Debian that we can install and just does that?)

Since USB and PS/2 mice share the same device since kernel 2.6, that part of xorg.conf doesn't strictly need to be detected, same for the keyboard (alps and synaptic touchpads can be easily detected) and X.org can use the screen's ddc info although it's not always reliable.

It can boil down to just detecting the video chipset: something like this, that uses PCI IDs from discover1-data.

It can also become a lot easier with X.org's own ddc detection, which almost boils down to configuring input devices and selecting the video driver. If I understand Daniel Stone correctly, X.org will soon improve its detection routines (fail safe X (auto-)configuration) as well in X.org 7.3.

xresprobe is in debian: it's pretty similar to ddcxinfo-kanotix, both forked off RedHat's kudzu package - and all fail miserably on amd64. That's why ddcxinfo has a fallback to 1024*768 @75 Hz which "always works (+manual overrides)".

Posted Sat Jun 6 00:57:39 2009 Tags:

First pratical lesson

Notes after today's training session.

Small index of most used shell commands:

  • ls - list directory contents
  • cp - copy files and directories
  • mv - move (rename) files
  • rm - remove files or directories
  • find - search for files in a directory hierarchy
  • cat - concatenate files and print on the standard output
  • more - file perusal filter for crt viewing
  • less - opposite of more (quit with 'q')
  • cd - Change the current directory to DIR. (use "help cd" instead of "man cd")
  • mkdir - make directories
  • rmdir - remove empty directories

Small index of commands useful for combining in pipelines:

  • grep, egrep, fgrep, rgrep - print lines matching a pattern
  • tail - output the last part of files
  • head - output the first part of files
  • sort - sort lines of text files
  • uniq - report or omit repeated lines
  • sed - stream editor
  • wc - print the number of newlines, words, and bytes in files

Problems found during the lesson:

  • You set the system default locale to Amharic, and the gdm login will be in Amharic input mode. We didn't find out how to switch it back to input roman characters. Right click on the input field to set the input method doesn't work. Since usernames are not in Amharic, you're locked out.
  • So you CTRL+ALT+F1, login and try dpkg-reconfigure locales. On Ubuntu Dapper, it does not work anymore.
  • So you dig and dig and dig and finally find that you can force a locale in /etc/default/gdm (but not in /etc/gdm/locale.conf, nor in /etc/gdm/gdm.conf).
  • Then the internet works for a bit and you look up how to reconfigure locales in Ubuntu. Turns out you have to use localeconf, which is not installed by default, is not in universe and thus not on the CDs, and needs to be downloaded from the Internet.
  • The Ubuntu wiki is all on https, which defeats any attempt of proxy caching.
  • An Internet proxy needs to be configured 3 times: in Gnome, in Firefox and in Synaptic (well, apt). This is especially tricky when you forgot to setup the proxy in Synaptic and seemingly unrelated applications fail, like the Ubuntu language selector, which internally invokes the package manager to download missing langpacks.
  • Some short descriptions in the NAME section of manpages are hard to understand, or wrong. Noted on apt-get, apt-cache and less. Top prize goes to apt-cache:

     NAME
            apt-cache - APT package handling utility -- cache manipulator
     DESCRIPTION
            [...] apt-cache does not manipulate the state of the system but
            does provide operations to search and generate interesting output
            from the package metadata. [...]
    

    So apt-cache is a manipulator that doesn't manipulate. A possible improvement can be "query the APT package cache".

  • The language selector in Ubuntu Breezy doesn't really exit and keeps the package database locked. This seems to be fixed in Dapper, and probably had been fixed in some Breezy update. System updates here are a problem: my Dapper (with some Universe things in it) wanted to download more than 120Mb of data, and the Uni network was giving me 14Kbps. It's been a nice opportunity to teach about fuser -uva and kill.
  • dict, squid and many other packages from 'main' are not on the normal Ubuntu CDs: is there an easy way to build a CD with them? Or do Ubuntu CDs with extra packages already exist? I'll have to find out.
  • cupsys has documentation outside of /usr/share/doc, in /usr/share/cups/doc-root.
  • man works on all commands, except cd, which is an internal shell command and thus needs help instead of man. I should remember to ponder about autogenerating manpages from help output.
  • Is there an index-like manpage with a list of the core Unix commands and their short descriptions? It there's not, it's easy to generate:

     #!/bin/sh
     DIR=${1:-"/bin"}
     (
     find $DIR | while read FILE
     do
         if [ -x $FILE ] && ! [ -d $FILE ]
         then
             LANG=C COLUMNS=2000 man `basename $FILE` | \
                      grep ^SYNOPSIS -B 100 | grep ^NAME -A 100 | \
                      tail -n +2 | head -n +2 | \
                      grep -v '^[ \t]*$'
         fi
     done
     ) | sort | uniq | sed 's/^ \+//'
    

    Try running it on /bin and /sbin: it's great!. Also, since it doesn't redirect stderr, it nicely exposes a number of manpage problems.

Lots of bugs to report when I come home: from here it'll take ages, and lots of money on the hotel internet connection, and some are Ubuntu-specific so I'd need to do everything online with Malone.

As usual, teaching is one of the best ways to find bugs.

I propose an Etch training session a month before release.

Other things to do:

  • Find more info about that Wikipedia live CD with Wikipedia browsable without the Internet.
  • Make a collection of Free technical E-books: even those Indian low-cost book editions are too expensive here, so E-books mean a lot.

Update: Matt Zimmerman writes:

I read your blog entry at http://www.enricozini.org/blog/eng/second-day-in-addis and wanted to respond as follows:

  • localeconf is not the standard way to configure locales in Ubuntu; what documentation told you that? It's an unsupported package from Progeny. If what you wanted was to set the system default locale from the command line, editing /etc/environment is probably the best way.

  • I suggest filing a bug report at <https://launchpad.net/products/ubuntu-website about the HTTPS issue>; I don't think it's necessary for the entire wiki to be HTTPS, only authentication.

  • Synaptic may be able to use the GNOME proxy settings without introducing undesirable dependencies; please file a wishlist bug

  • dict, squid and other packages from main are not on the Ubuntu CDs because there is no space. The DVD contains these packages.

  • The cupsys documentation bug was quite likely inherited from Debian and should be reported there

  • You can file bugs in Malone via email; this has been possible for a long time now. Please don't reinforce this misconception.

    https://help.launchpad.net/UsingMaloneEmail

Update:

Posted Sat Jun 6 00:57:39 2009 Tags:

Fixing problems after upgrade to Dapper

Laptop: Asus M3Ae

Problem: Can't mount root partition because of various ACPI errors. Breezy kernel works.

Solution:

1) boot with old kernel 2) sudo echo "libata noacpi=1" >> /etc/mkinitramfs/modules 3) sudo mv /boot/initrd.img-2.6.15-25-686 /boot/initrd.img-2.6.15-25-686.backup 4) mkinitramfs -o /boot/initrd.img-2.6.15-25-686 2.6.15-25-686

Thanks: Matthew Garrett

Posted Sat Jun 6 00:57:39 2009 Tags:

Live CD on a removable disk, the Debian way

In [live-cd-on-removable-disk] at some point I wrote:

Enrico's note: do we have anything in Debian that we can install and just does that?

Here are the answers:

Sven Mueller writes:

Well, Enrico, a tool I really grew fond of, which auto-configures X on Debian systems is xdebconfigurator, it lacks being auto-run on each system start, which I consider a feature on normal systems, but for your proposed usage (i.e. a portable USB-storage based Debian system), it would certainly be the right thing.

Essentially, it never failed on me. Except for VMware virtual machines, where all it did wrong was that it proposed too high resolutions which resulted from my dual-screen Windows setup I ran VMware on. You might want to give it a try.

Tollef Fog Heen writes:

I added the support in casper for doing this almost a year ago and it has saved me lots of debugging time. Booting the live CD that way is almost as fast as booting an installed system. If you couple this with using the persistent storage support in casper, you can get the configure-on-boot support together with persistency.

In a later update, slh is quited saying that xresprobe doesn't work on AMD64. This is wrong, I wrote that support based on code by Matthew Garret a little more than nine months ago. I wouldn't recommend incorporating it in new-written code, but rather use libx86

And finally, Marco Amadori writes:

Without needing to look for tools external to Debian, there is already the Debian Live software in sid: live-package, that creates a live system, and casper, that generates an initramfs that can configure a Debian system on the fly.

So far there is no hard disk target for live-package, but the "Iso" target can already do the job quite well. At boot time, Casper's initramfs scans all the block devices, so it works also for USB keys and hard drives.

To obtain a hard drive image, you just need to invoke "make-live" with the options to have the required software, then copy the content of the iso (or of the directory ./debian-live/binary) on a partition and install the boot loader.

This is what the future "HD" target of live-package will do; so far it can only build ISO and Netboot images.

Posted Sat Jun 6 00:57:39 2009 Tags:
Posted Sat Jun 6 00:57:39 2009