Enrico's blog

Last 10 blog posts

2016-11-15 13:01:39+01:00

Software quality in 2016

Ansible's default output, including the stderr of failed commands, is JSON encoded, which makes reading Jenkins' output hard.

Ansible however has Callback plugins that could be used. In that page it says:

Ansible comes with a number of callback plugins that you can look at for examples. These can be found in lib/ansible/plugins/callback.

That is a link to a git repo with just a pile of Python sources and no, say README.md index to what they do. Hopefully they have some docstring with a short description of what they do? no.

Actually, some do, but just because someone copypasted the default one and didn't even bother removing its docstring.

frustration

debian eng pdo rant
2016-11-09 10:10:42+01:00

On SPF

I woke up this morning with some Django server error mails in my inbox:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 9: ordinal not in range(128)
…
 'REMOTE_USER': u'…-guest@users.alioth.debian.org',

I did what one does in cases like these, I tried to fix the bug and mailed …-guest@users.alioth.debian.org asking them to try again and let me know if it works.

I get a bounce:

  <Actual user's email>
    (generated from …-guest@users.alioth.debian.org)
    SMTP error from remote mail server after MAIL FROM:<enrico@enricozini.org> SIZE=3948:
    host … […]: 550 Please see http://www.openspf.org/Why?id=enrico%40enricozini.org&ip=2001%3a41c8%3a1000%3a21%3a%3a21%3a21&receiver=bq :
    Reason: mechanism

I resent the mail to the actual user's address, and it went through. Job done, at least until they get back to me telling me that my fix didn't work.

Lessons learnt:

debian eng pdo sw
2016-11-02 16:54:06+01:00

schroot connector for ansible

I accidentally made an ansible connector plugin for schroot. I haven't even used ansible yet, so I have no idea what I am doing.

You can choose the chroot type by setting something like schroot_type: jessie in your variables.

You can install this locally as connection_plugins/schroot.py.

# Based on chroot.py (c) 2013, Maykel Moya <mmoya@speedyrails.com>
# Based on chroot.py (c) 2015, Toshio Kuratomi <tkuratomi@ansible.com>
# (c) 2016, Enrico Zini <enrico@debian.org>
#
# This is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Ansible is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

import distutils.spawn
import os
import os.path
import pipes
import subprocess
import traceback

from ansible import constants as C
from ansible.errors import AnsibleError
from ansible.plugins.connection import ConnectionBase, BUFSIZE
from ansible.module_utils.basic import is_executable
from ansible.utils.unicode import to_bytes

try:
    from __main__ import display
except ImportError:
    from ansible.utils.display import Display
    display = Display()


class Connection(ConnectionBase):
    ''' Local chroot based connections '''

    transport = 'schroot'
    has_pipelining = True
    # su currently has an undiagnosed issue with calculating the file
    # checksums (so copy, for instance, doesn't work right)
    # Have to look into that before re-enabling this
    become_methods = frozenset(C.BECOME_METHODS).difference(('su',))

    def __init__(self, play_context, new_stdin, *args, **kwargs):
        super(Connection, self).__init__(play_context, new_stdin, *args, **kwargs)

        self.chroot = self._play_context.remote_addr

        #if os.geteuid() != 0:
            #raise AnsibleError("chroot connection requires running as root")

        # we're running as root on the local system so do some
        # trivial checks for ensuring 'host' is actually a chroot'able dir
        #if not os.path.isdir(self.chroot):
        #    raise AnsibleError("%s is not a directory" % self.chroot)

        #chrootsh = os.path.join(self.chroot, 'bin/sh')
        ## Want to check for a usable bourne shell inside the chroot.
        ## is_executable() == True is sufficient.  For symlinks it
        ## gets really complicated really fast.  So we punt on finding that
        ## out.  As long as it's a symlink we assume that it will work
        #if not (is_executable(chrootsh) or (os.path.lexists(chrootsh) and os.path.islink(chrootsh))):
        #    raise AnsibleError("%s does not look like a chrootable dir (/bin/sh missing)" % self.chroot)

        self.chroot_cmd = distutils.spawn.find_executable('schroot')
        if not self.chroot_cmd:
            raise AnsibleError("chroot command not found in PATH")

        self.chroot_id = "session:" + self.chroot
        self.chroot_type = "stable"
        existing = subprocess.check_output([self.chroot_cmd, "--list", "--all"])
        self.chroot_exists = False
        for line in existing.splitlines():
            if line == self.chroot_id:
                self.chroot_exists = True

    def set_host_overrides(self, host, hostvars=None):
        super(Connection, self).set_host_overrides(host, hostvars)
        self.chroot_type = hostvars.get("schroot_type", self.chroot_type)

    def _connect(self):
        ''' connect to the chroot; nothing to do here '''
        super(Connection, self)._connect()
        if not self._connected:
            if not self.chroot_exists:
                self.chroot_id = subprocess.check_output([self.chroot_cmd, "-b", "-c", self.chroot_type, "-n", self.chroot]).strip()
                subprocess.check_call([self.chroot_cmd, "-r", "-c", self.chroot_id, "-u", "root", "--", "apt-get", "update"])
                subprocess.check_call([self.chroot_cmd, "-r", "-c", self.chroot_id, "-u", "root", "--", "apt-get", "-y", "install", "python"])

            display.vvv("THIS IS A LOCAL CHROOT DIR", host=self.chroot)
            self._connected = True

    def _buffered_exec_command(self, cmd, stdin=subprocess.PIPE):
        ''' run a command on the chroot.  This is only needed for implementing
        put_file() get_file() so that we don't have to read the whole file
        into memory.

        compared to exec_command() it looses some niceties like being able to
        return the process's exit code immediately.
        '''
        executable = C.DEFAULT_EXECUTABLE.split()[0] if C.DEFAULT_EXECUTABLE else '/bin/sh'
        local_cmd = [self.chroot_cmd, "-r", "-c", self.chroot_id, "-u", "root", "--", executable, '-c', cmd]

        display.vvv("EXEC %s" % (local_cmd), host=self.chroot)
        local_cmd = [to_bytes(i, errors='strict') for i in local_cmd]
        p = subprocess.Popen(local_cmd, shell=False, stdin=stdin,
                stdout=subprocess.PIPE, stderr=subprocess.PIPE)

        return p

    def exec_command(self, cmd, in_data=None, sudoable=False):
        ''' run a command on the chroot '''
        super(Connection, self).exec_command(cmd, in_data=in_data, sudoable=sudoable)

        p = self._buffered_exec_command(cmd)

        stdout, stderr = p.communicate(in_data)
        return (p.returncode, stdout, stderr)

    def _prefix_login_path(self, remote_path):
        ''' Make sure that we put files into a standard path

            If a path is relative, then we need to choose where to put it.
            ssh chooses $HOME but we aren't guaranteed that a home dir will
            exist in any given chroot.  So for now we're choosing "/" instead.
            This also happens to be the former default.

            Can revisit using $HOME instead if it's a problem
        '''
        if not remote_path.startswith(os.path.sep):
            remote_path = os.path.join(os.path.sep, remote_path)
        return os.path.normpath(remote_path)

    def put_file(self, in_path, out_path):
        ''' transfer a file from local to chroot '''
        super(Connection, self).put_file(in_path, out_path)
        display.vvv("PUT %s TO %s" % (in_path, out_path), host=self.chroot)

        out_path = pipes.quote(self._prefix_login_path(out_path))
        try:
            with open(to_bytes(in_path, errors='strict'), 'rb') as in_file:
                try:
                    p = self._buffered_exec_command('dd of=%s bs=%s' % (out_path, BUFSIZE), stdin=in_file)
                except OSError:
                    raise AnsibleError("chroot connection requires dd command in the chroot")
                try:
                    stdout, stderr = p.communicate()
                except:
                    traceback.print_exc()
                    raise AnsibleError("failed to transfer file %s to %s" % (in_path, out_path))
                if p.returncode != 0:
                    raise AnsibleError("failed to transfer file %s to %s:\n%s\n%s" % (in_path, out_path, stdout, stderr))
        except IOError:
            raise AnsibleError("file or module does not exist at: %s" % in_path)

    def fetch_file(self, in_path, out_path):
        ''' fetch a file from chroot to local '''
        super(Connection, self).fetch_file(in_path, out_path)
        display.vvv("FETCH %s TO %s" % (in_path, out_path), host=self.chroot)

        in_path = pipes.quote(self._prefix_login_path(in_path))
        try:
            p = self._buffered_exec_command('dd if=%s bs=%s' % (in_path, BUFSIZE))
        except OSError:
            raise AnsibleError("chroot connection requires dd command in the chroot")

        with open(to_bytes(out_path, errors='strict'), 'wb+') as out_file:
            try:
                chunk = p.stdout.read(BUFSIZE)
                while chunk:
                    out_file.write(chunk)
                    chunk = p.stdout.read(BUFSIZE)
            except:
                traceback.print_exc()
                raise AnsibleError("failed to transfer file %s to %s" % (in_path, out_path))
            stdout, stderr = p.communicate()
            if p.returncode != 0:
                raise AnsibleError("failed to transfer file %s to %s:\n%s\n%s" % (in_path, out_path, stdout, stderr))

    def close(self):
        ''' terminate the connection; nothing to do here '''
        super(Connection, self).close()
        #subprocess.check_command([self.chroot_cmd, "-e", "-c", self.chroot_id])
        self._connected = False
debian eng pdo sw
2016-11-01 00:00:00+02:00

Links for November 2016

So You Want To Learn Physics... [archive]
«This post is a condensed version of what I've sent to people who have contacted me over the years, outlining what everyone needs to learn in order to really understand physics.»
Operation Tamarisk [archive]
«Operation Tamarisk was a Cold War-era operation run by the military intelligence services of the U.S., U.K., and France through their military liaison missions in East Germany, that gathered discarded paper, letters, and garbage from Soviet trash bins and military maneuvers, including used toilet paper.»
Mortara case
Of how in Bologna, where I live, in the 1850s/1860s, when my grand-grand-granddad lived, the Papal State seized a child from a Jewish family «on the basis of a former servant's testimony that she had administered emergency baptism to the boy when he fell sick as an infant.»
eng links pdo
2016-10-30 19:10:25+01:00

Bremen Freimarkt Parade

Look! They are having a parade with people dressed like Debian releases!

Bremen Freimarkt Parade

debian eng pdo
2016-10-18 10:25:55+02:00

debtags and aptitude forget-new

I like to regularly go through the new packages section in aptitude to see what interesting new packages entered testing, but recently that joyful moment got less joyful for me because of a barrage of obscurely named packages.

I have just realised that aptitude forget-new supports search patterns, and that brought back the joy.

I put this in a script that I run before looking for new packages in aptitude:

aptitude forget-new '?tag(field::biology)
                   | ?tag(devel::lang:ruby)
                   | ?tag(devel::lang:perl)
                   | ?tag(role::shared-lib)
                   | ?tag(suite::openstack)
                   | ?tag(implemented-in::php)
                   | ~n^node-'

The actual content of the search pattern is purely a matter of taste.

I'm happy to see how debtags becomes quite useful here, to keep my own user experience manageable as the size of Debian keeps growing.

Update: pabs suggested to use apt post-invoke hooks. For example:

        $ cat /etc/apt/apt.conf.d/99forget-new
        APT::Update::Post-Invoke { "aptitude forget-new '~sdebug'"; };
debian eng pdo
2016-10-01 00:00:00+02:00

Links for October 2016

Inside the Federal Bureau Of Way Too Many Guns [archive]
«There's no telling how many guns we have in America—and when one gets used in a crime, no way for the cops to connect it to its owner. The only place the police can turn for help is a Kafkaesque agency in West Virginia, where, thanks to the gun lobby, computers are illegal and detective work is absurdly antiquated. On purpose. Thing is, the geniuses who work there are quietly inventing ways to do the impossible.»
What to do when you see harassment [archive]
«This is an illustrated guide I made as part of my co-admining work at The Middle Eastern Feminist on Facebook! It will be published there shortly. The technique that is displayed here is a genuine one used in psychology - I forgot the name and couldn’t find it again so if you know about it, feel free to tell me! Some could say: “Yes but you can use that technique for instances of harassment other than Islamophobic attacks!”, and my reply is: Sure! Please do so, it also works for other “types” of harassment of a lone person in a public space!!» (buzzfeed article) (article on noncomplementary behaviour)
Stop Stealing Dreams [archive]
«Over the last ten years, I’ve written more than a dozen books about how our society is being fundamentally changed by the impact of the internet and the connection economy. [...] In this manifesto, I’m going to argue that top-down industrialized schooling is just as threatened, and for very good reasons. Scarcity of access is destroyed by the connection economy, at the very same time the skills and attitudes we need from our graduates are changing.»
English Adjectives Order
«When we group adjectives together there is a general (sometimes flexible) rule for the position of each type of adjective». More details. I like this as a reminder that I don't need to learn gramar in order to use it. Which is why I speak much better German after fiddling occasionally on Duolingo and buying schinkenzwiebelmettwürst at the wochenmarkt, than Latin after insulting my intelligence by pointlessly studying only its grammar for 5 long years in high school.
there will be a moment when you realize you are more grown up than your parents are… [archive]
Short piece of prose on Tumblr that I related to a lot.
eng links pdo
2016-09-10 09:47:03+02:00

Dreaming of being picked

From "Stop stealing dreams":

«Settling for the not-particularly uplifting dream of a boring, steady job isn’t helpful. Dreaming of being picked — picked to be on TV or picked to play on a team or picked to be lucky — isn’t helpful either. We waste our time and the time of our students when we set them up with pipe dreams that don’t empower them to adapt (or better yet, lead) when the world doesn’t work out as they hope.

The dreams we need are self-reliant dreams. We need dreams based not on what is but on what might be. We need students who can learn how to learn, who can discover how to push themselves and are generous enough and honest enough to engage with the outside world to make those dreams happen.»

This made me think that I know many hero stories based on "the chosen", like Matrix, like most superheros getting powers either from some entity chosing them for it, or from chance.

I have a hard time thinking of a superhero who becomes one just by working hard at acquiring and honing their skills: I can only think of Batman and Ironman, and they start off as super rich.

If I think of people who start from scratch as commoners and work hard to become exceptional, in the standard superhero narrative, I can only think of supervillains.

Scary.

It makes me feel culturally biased into thinking that a common person cannot be trusted to act responsibly, and that only the rich, the chosen and the aristocrats can.

As a bias it may serve the rich and the aristocrats, but I don't think it serves society as a whole.

eng life pdo
2016-09-01 00:00:00+02:00

Links for September 2016

A Few Useful Mental Tools from Richard Feynman [archive]

«These tricks show Feynman taking the method of thought he learned in pure science and applying it to the more mundane topics most of us have to deal with every day.»

Pasta [archive]

A comprehensive introduction to pasta, to keep at hand in case I meet someone who has little familiarity with it.

MPTP: One Designer Drug and Serendipity [archive]

Abstract: Through an unlikely series of coincidences and fortunate accidents, the development of Parkinson’s disease in several illicit drug users was traced to their use of a meperidine analog contaminated with 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine (MPTP). The discovery of a chemical capable of producing animal models of the disease has revitalized research efforts and resulted in important new information. The serendipitous finding also prompted consideration of what changes seem advisable if designer drugs are to be dealt with more efficaciously.

The Debunking Handbook: now freely available for download

The Debunking Handbook, a guide to debunking misinformation, is now freely available to download. Although there is a great deal of psychological research on misinformation, there's no summary of the literature that offers practical guidelines on the most effective ways of reducing the influence of myths.

Faulty neon light jams radio appliances [archive]

«An apparent interference source began plaguing wireless vehicle key fobs, cell phones, and other wireless electronics. Key fob owners found they could not open or start their vehicles remotely until their vehicles were towed at least a block away, nor were they able to call for help on their cell phones when problems occurred»

Calvin & Muad'Dib

Calvin & Hobbes with text taken from Frank Herbert's Dune. It's been around since 2013 and I consistently found it moving and deep.

When Birds Attack - Bike Helmet Hacks [archive]

Australian magpies attacking cyclists has propted several creative adaptations, including attaching an afro wig to the bike helmet.

eng links pdo
2016-08-01 00:00:00+02:00

Links for August 2016

First post with the new link collection feature of staticsite!

Heavy Metal and Natural Language Processing [archived]

Natural language processing and Metal lyrics, including the formula for the "metalness" of a word and a list of the most and least metal words.

Confirming all use of an SSH agent [archived]

«For a long time I’ve wanted an ssh-agent setup that would ask me before every use, so I could slightly more comfortably forward authentication over SSH without worrying that my session might get hijacked somewhere at the remote end (I often find myself wanting to pull authenticated git repos on remote hosts). I’m at DebConf this week, which is an ideal time to dig further into these things, so I did so today. As is often the case it turns out this is already possible, if you know how.»

Why We Don’t Report It [archived]

«“Why don’t you report it?” It’s up there on every list I’ve seen of things you shouldn’t say to sexual assault survivors, yet I keep hearing it…»

Voltron, an extensible debugger UI toolkit written in Python

Multi-panel display built from various gdb outputs.

Notmuch, offlineimap and Sieve setup [archived]

Nice description of a notmuch+offlineimap+sieve setup, for when I feel like rethinking my email setup.

Wikipedia:Unusual articles

An endless source of weird and wonderful.

ZERO: no linked HIV transmissions [archived]

«The results provide a dataset to question whether transmission with an undetectable viral load is actually possible. They should help normalise HIV and challenge stigma and discrimination.»

TV pickup

Someone once in the UK told me that it was a big enough problem that so many people turn on their electric kettles during the endtitles of Eastenders, that there's an employee in a hydro plant that needs to watch it to ramp up the power at the right time. I've finally found a wikipedia page about it.

Amazon isn't saying if Echo has been wiretapped [archived]

"We may never know if the feds have hijacked Amazon Echo. … In case you didn't know, Echo is an always-on device, which, when activated, can return search queries, as well as read audiobooks and report sports, traffic, and weather. It can even control smart home devices."

eng pdo