All entries

Another day in the life of a poor developer

try:
    # After Python 3.3
    from collections.abc import Iterable
except ImportError:
    # This has changed in Python 3.3 (why, oh why?), reinforcing the idea that
    # the best Python version ever is still 2.7, simply because upstream has
    # promised that they won't touch it (and break it) for at least 5 more
    # years.
    from collections import Iterable

import shlex
if hasattr(shlex, "quote"):
    # New in version 3.3.
    shell_quote = shlex.quote
else:
    # Available since python 1.6 but deprecated since version 2.7: Prior to Python
    # 2.7, this function was not publicly documented. It is finally exposed
    # publicly in Python 3.3 as the quote function in the shlex module.
    #
    # Except everyone was using it, because it was the only way provided by the
    # python standard library to make a string safe for shell use
    #
    # See http://stackoverflow.com/questions/35817/how-to-escape-os-system-calls-in-python
    import pipes
    shell_quote = pipes.quote

import shutil
if hasattr(shutil, "which"):
    # New in version 3.3.
    shell_which = shutil.which
else:
    # Available since python 1.6:
    # http://stackoverflow.com/questions/377017/test-if-executable-exists-in-python
    from distutils.spawn import find_executable
    shell_which = find_executable
Posted Fri Feb 27 12:02:33 2015 Tags:

Akonadi client example

After many failed attemps I have managed to build a C++ akonadi client. It has felt like one of the most frustrating programming experiences of my whole life, so I'm sharing the results hoping to spare others from all the suffering.

First thing first, akonadi client libraries are not in libakonadi-dev but in kdepimlibs5-dev, even if kdepimlibs5-dev does not show in apt-cache search akonadi.

Then, kdepimlibs is built with Qt4. If your application uses Qt5 (mine was) you need to port it back to Qt4 if you want to talk to Akonadi.

Then, kdepimlibs does not seem to support qmake and does not ship pkg-config .pc files, and if you want to use kdepimlibs your build system needs to be cmake. I ported by code from qmake to cmake, and now qtcreator wants me to run cmake by hand every time I change the CMakeLists.txt file, and it stopped allowing to add, rename or delete sources.

Finally, most of the code / build system snippets found on the internet seem flawed in a way or another, because the build toolchain of Qt/KDE applications has undergone several redesignins during time, and the network is littered with examples from different eras. The way to obtain template code to start a Qt/KDE project is to use kapptemplate. I have found no getting started tutorial on the internet that said "do not just copy the snippets from here, run kapptemplate instead so you get them up to date".

kapptemplate supports building an "Akonadi Resource" and an "Akonadi Serializer", but it does not support generating template code for an akonadi client. That left me with the feeling that I was dealing with some software that wants to be developed but does not want to be used.

Anyway, now an example of how to interrogate Akonadi exists as is on the internet. I hope that all the tears of blood that I cried this morning have not been cried in vain.

Posted Mon Feb 23 15:44:01 2015 Tags:

The wonders of missing documentation

Update: I have managed to build an example Akonadi client application.

I'm new here, I want to make a simple C++ GUI app that pops up a QCalendarWidget which my local Akonadi has appointments.

I open qtcreator, create a new app, hack away for a while, then of course I get undefined references for all Akonadi symbols, since I didn't tell the build system that I'm building with akonadi. Ok.

How do I tell the build system that I'm building with akonadi? After 20 minutes of frantic looking around the internet, I still have no idea.

There is a package called libakonadi-dev which does not seem to have anything to do with this. That page mentions everything about making applications with Akonadi except how to build them.

There is a package called kdepimlibs5-dev which looks promising: it has no .a files but it does have haders and cmake files. However, qtcreator is only integrated with qmake, and I would really like the handholding of an IDE at this stage.

I put something together naively doing just what looked right, and I managed to get an application that segfaults before main() is even called:

/*
 * Copyright © 2015 Enrico Zini <enrico@enricozini.org>
 *
 * This work is free. You can redistribute it and/or modify it under the
 * terms of the Do What The Fuck You Want To Public License, Version 2,
 * as published by Sam Hocevar. See the COPYING file for more details.
 */
#include <QDebug>

int main(int argc, char *argv[])
{
    qDebug() << "BEGIN";
    return 0;
}
QT       += core gui widgets
CONFIG += c++11

TARGET = wtf
TEMPLATE = app

LIBS += -lkdecore -lakonadi-kde

SOURCES += wtf.cpp

I didn't achieve what I wanted, but I feel like I achieved something magical and beautiful after all.

I shall now perform some haruspicy on those oscure cmake files to see if I can figure something out. But seriously, people?

Posted Mon Feb 23 11:36:18 2015 Tags:

Setting up Akonadi

Now that I have a CalDAV server that syncs with my phone I would like to use it from my desktop.

It looks like akonadi is able to sync with CalDAV servers, so I'm giving it a try.

First thing first is to give a meaning to the arbitrary name of this thing. Wikipedia says it is the oracle goddess of justice in Ghana. That still does not hint at all at personal information servers, but seems quite nice. Ok. I gave up with software having purpose-related names ages ago.

# apt-get install akonadi-server akonadi-backend-postgresql

Akonadi wants a SQL database as a backend. By default it uses MySQL, but I had enough of MySQL ages ago.

I tried SQLite but the performance with it is terrible. Terrible as in, it takes 2 minutes between adding a calendar entry and having it show up in the calendar. I'm fascinated by how Akonadi manages to use SQLite so badly, but since I currently just want to get a job done, next in line is PostgreSQL:

# su - postgres
$ createuser enrico
$ psql postgres
postgres=# alter user enrico createdb;

Then as enrico:

$ createdb akonadi-enrico
$ cat <<EOT > ~/.config/akonadi/akonadiserverrc
[%General]
Driver=QPSQL

[QPSQL]
Name=akonadi-enrico
StartServer=false
Host=
Options=
ServerPath=
InitDbPath=

I can now use kontact to connect Akonadi to my CalDAV server and it works nicely, both with calendar and with addressbook entries.

KDE has at least two clients for Akonadi: Kontact, which is a kitchen sink application similar to Evolution, and KOrganizer, which is just the calendar and scheduling component of Kontact.

Both work decently, and KOrganizer has a pretty decent startup time. I now have a usable desktop PIM application that is synced with my phone. W00T!

Next step is to port my swift little calendar display tool to use Akonadi as a back-end.

Posted Tue Feb 17 15:34:55 2015 Tags:

seat-inspect

Four months ago I wrote this somewhere:

Seeing a DD saying "this new dbus stuff scares me" would make most debian users scared. Seeing a DD who has an idea of what is going on, and who can explain it, would be an interesting and exciting experience.

So, let's be exemplary, competent and patient. Or at least, competent. Some may like or not like the changes, but do we all understand what is going on? Will we all be able to support our friends and customers running jessie?

I confess that although I understand the need for it, I don't feel competent enough to support systemd-based machines right now.

So, are we maybe in need of help, cheat sheets, arsenals of one-liners, diagnostic tools?

Maybe a round of posts on -planet like "one debian package a day" but with new features that jessie will have, and how to understand them and take advantage of them?

That was four months ago. In the meantime, I did some work, and it got better for me.

Yesterday, however, I've seen an experienced Linux person frustrated because the shutdown function of the desktop was doing nothing whatsoever. Today I found John Goerzen's post on planet.

I felt like some more diagnostic tools were needed, so I spent the day making seat-inspect.

seat-inspect tries to make the status of the login/seat system visible, to help with understanding and troubleshooting.

The intent of running the code is to have an overview of the system status, both to see what the new facilities are about, and to figure out if there is something out of place.

The intent of reading the code is to have an idea of how to use these facilities: the code has been written to be straightforward and is annotated with relevant bits from the logind API documentation.

seat-inspect is not a finished tool, but a starting point. I put it on github hoping that people will fork it and add their own extra sanity checks and warnings, so that it can grow into a standard thing to run if a system acts weird.

As it is now, it should be able to issue warnings if some bits are missing for network-manager or shutdown functions to work correctly. I haven't really tested that, though, because I don't have a system at hand where they are currently not working fine.

Another nice thing of it is that when running seat-inspect -v you get a dump of what logind/consolekit think about your system. I found it an interesting way to explore the new functionalities that we recently grew. The same can be done, and in more details, with loginctl calls, but I lacked a summary.

After writing this I feel a bit more competent, probably enough to sit at somebody's computer and poke into loginctl bits. I highly recommend the experience.

Posted Tue Feb 10 18:06:43 2015 Tags:

Mozilla marketplace facepalm

This made me sad.

My view, which didn't seem to be considered in that discussion, is that people concerned about software freedom and security are likely to stay the hell away from such an app market and its feedback forms.

Also, that thread made me so sad about the state of that developer community that I seriously do not feel like investing energy into going through the hoops of getting an account in their bugtracker to point this out.

Sigh.

Posted Fri Jan 23 15:13:16 2015 Tags:

Playing with python, terminfo and command output

I am experimenting with showing progress on the terminal for a subcommand that is being run, showing what is happening without scrolling away the output of the main program, and I came out with this little toy. It shows the last X lines of a subcommand output, then gets rid of everything after the command has ended.

Usability-wise, it feels like a tease to me: it looks like I'm being shown all sorts of information then they are taken away from me before I managed to make sense of them. However, I find it cute enough to share:

#!/usr/bin/env python3
#coding: utf-8
# Copyright 2015 Enrico Zini <enrico@enricozini.org>.  Licensed under the terms
# of the GNU General Public License, version 2 or any later version.

import argparse
import fcntl
import select
import curses
import contextlib
import subprocess
import os
import sys
import collections
import shlex
import shutil
import logging

def stream_output(proc):
    """
    Take a subprocess.Popen object and generate its output, line by line,
    annotated with "stdout" or "stderr". At process termination it generates
    one last element: ("result", return_code) with the return code of the
    process.
    """
    fds = [proc.stdout, proc.stderr]
    bufs = [b"", b""]
    types = ["stdout", "stderr"]
    # Set both pipes as non-blocking
    for fd in fds:
        fcntl.fcntl(fd, fcntl.F_SETFL, os.O_NONBLOCK)
    # Multiplex stdout and stderr with different prefixes
    while len(fds) > 0:
        s = select.select(fds, (), ())
        for fd in s[0]:
            idx = fds.index(fd)
            buf = fd.read()
            if len(buf) == 0:
                fds.pop(idx)
                if len(bufs[idx]) != 0:
                    yield types[idx], bufs.pop(idx)
                types.pop(idx)
            else:
                bufs[idx] += buf
                lines = bufs[idx].split(b"\n")
                bufs[idx] = lines.pop()
                for l in lines:
                    yield types[idx], l
    res = proc.wait()
    yield "result", res

@contextlib.contextmanager
def miniscreen(has_fancyterm, name, maxlines=3, silent=False):
    """
    Show the output of a process scrolling in a portion of the screen.

    has_fancyterm: true if the terminal supports fancy features; if false, just
    write lines to standard output

    name: name of the process being run, to use as a header

    maxlines: maximum height of the miniscreen

    silent: do nothing whatsoever, used to disable this without needing to
            change the code structure

    Usage:
        with miniscreen(True, "my process", 5) as print_line:
            for i in range(10):
                print_line(("stdout", "stderr")[i % 2], "Line #{}".format(i))
    """
    if not silent and has_fancyterm:
        # Discover all the terminal control sequences that we need
        output_normal = str(curses.tigetstr("sgr0"), "ascii")
        output_up = str(curses.tigetstr("cuu1"), "ascii")
        output_clreol = str(curses.tigetstr("el"), "ascii")
        cols, lines = shutil.get_terminal_size()
        output_width = cols

        fg_color = (curses.tigetstr("setaf") or
                    curses.tigetstr("setf") or "")
        sys.stdout.write(str(curses.tparm(fg_color, 6), "ascii"))

        output_lines = collections.deque(maxlen=maxlines)

        def print_lines():
            """
            Print the lines in our buffer, then move back to the beginning
            """
            sys.stdout.write("{} progress:".format(name))
            sys.stdout.write(output_clreol)
            for msg in output_lines:
                sys.stdout.write("\n")
                sys.stdout.write(msg)
                sys.stdout.write(output_clreol)
            sys.stdout.write(output_up * len(output_lines))
            sys.stdout.write("\r")

        try:
            print_lines()

            def _progress_line(type, line):
                """
                Print a new line to the miniscreen
                """
                # Add the new line to our output buffer
                msg = "{} {}".format("." if type == "stdout" else "!", line)
                if len(msg) > output_width - 4:
                    msg = msg[:output_width - 4] + "..."
                output_lines.append(msg)
                # Update the miniscreen
                print_lines()

            yield _progress_line

            # Clear the miniscreen by filling our ring buffer with empty lines
            # then printing them out
            for i in range(maxlines):
                output_lines.append("")
            print_lines()
        finally:
            sys.stdout.write(output_normal)
    elif not silent:
        def _progress_line(type, line):
            print("{}: {}".format(type, line))
        yield _progress_line
    else:
        def _progress_line(type, line):
            pass
        yield _progress_line

def run_command_fancy(name, cmd, env=None, logfd=None, fancy=True, debug=False):
    quoted_cmd = " ".join(shlex.quote(x) for x in cmd)
    log.info("%s running command %s", name, quoted_cmd)
    if logfd: print("runcmd:", quoted_cmd, file=logfd)

    # Run the script itself on an empty environment, so that what was
    # documented is exactly what was run
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)

    with miniscreen(fancy, name, silent=debug) as progress:
        stderr = []
        for type, val in stream_output(proc):
            if type == "stdout":
                val = val.decode("utf-8")
                if logfd: print("stdout:", val, file=logfd)
                log.debug("%s stdout: %s", name, val)
                progress(type, val)
            elif type == "stderr":
                val = val.decode("utf-8")
                if logfd: print("stderr:", val, file=logfd)
                stderr.append(val)
                log.debug("%s stderr: %s", name, val)
                progress(type, val)
            elif type == "result":
                if logfd: print("retval:", val, file=logfd)
                log.debug("%s retval: %d", name, val)
                retval = val

    if retval != 0:
        lastlines = min(len(stderr), 5)
        log.error("%s exited with code %s", name, retval)
        log.error("Last %d lines of standard error:", lastlines)
        for line in stderr[-lastlines:]:
            log.error("%s: %s", name, line)

    return retval


parser = argparse.ArgumentParser(description="run a command showing only a portion of its output")
parser.add_argument("--logfile", action="store", help="specify a file where the full execution log will be written")
parser.add_argument("--debug", action="store_true", help="debugging output on the terminal")
parser.add_argument("--verbose", action="store_true", help="verbose output on the terminal")
parser.add_argument("command", nargs="*", help="command to run")
args = parser.parse_args()

if args.debug:
    loglevel = logging.DEBUG
elif args.verbose:
    loglevel = logging.INFO
else:
    loglevel = logging.WARN
logging.basicConfig(level=loglevel, stream=sys.stderr)
log = logging.getLogger()

fancy = False
if not args.debug and sys.stdout.isatty():
    curses.setupterm()
    if curses.tigetnum("colors") > 0:
        fancy = True

if args.logfile:
    logfd = open("output.log", "wt")
else:
    logfd = None

retval = run_command_fancy("miniscreen example", args.command, logfd=logfd)

sys.exit(retval)
Posted Wed Jan 21 11:13:31 2015 Tags:

Archive of all entries