Enrico's blog

Python output buffering

Here's a little toy program that displays a message like a split-flap display:

#!/usr/bin/python3

import sys
import time

def display(line: str):
    cur = '0' * len(line)
    while True:
        print(cur, end="\r")
        if cur == line:
            break
        time.sleep(0.09)
        cur = "".join(chr(min(ord(c) + 1, ord(oc))) for c, oc in zip(cur, line))
    print()

message = " ".join(sys.argv[1:])
display(message.upper())

This only works if the script's stdout is unbuffered. Pipe the output through cat, and you get a long wait, and then the final string, without the animation.

What is happening is that since the output is not going to a terminal, optimizations kick in that buffer the output and send it in bigger chunks, to make processing bulk I/O more efficient.

I haven't found a good introductory explanation of buffering in Python's documentation. The details seem to be scattered in the io module documentation and they mostly assume that one is already familiar with concepts like unbuffered, line-buffered or block-buffered. The libc documentation has a good quick introduction that one can read to get up to speed.

Controlling buffering in Python

In Python, one can force a buffer flush with the flush() method of the output file descriptor, like sys.stdout.flush(), to make sure pending buffered output gets sent.

Python's print() function also supports flush=True as an optional argument:

    print(cur, end="\r", flush=True)

If one wants to change the default buffering for a file descriptor, since Python 3.7 there's a convenient reconfigure() method, which can reconfigure line buffering only:

sys.stdout.reconfigure(line_buffering=True)

Otherwise, the technique is to reassign sys.stdout to something that has the behaviour one wants (code from this StackOverflow thread):

import io
# Python 3, open as binary, then wrap in a TextIOWrapper with write-through.
sys.stdout = io.TextIOWrapper(open(sys.stdout.fileno(), 'wb', 0), write_through=True)

If one needs all this to implement a progressbar, one should make sure to have a look at the progressbar module first.

pdo python eng

Stallman

When I hear Stallman saying "and I'm not planning to resign a second time", the only thing I can see is a dangerous person making a power move. I'll be wary of FSF from now on.

For this and other reasons, I have signed this open letter.

pdo debian life eng

nspawn-runner and ansible

This post is part of a series about trying to setup a gitlab runner based on systemd-nspawn. I published the polished result as nspawn-runner on GitHub.

New goal: make it easier to configure and maintain chroots. For example, it should be possible to maintain a rolling testing or sid chroot without the need to manually log into it to run apt upgrade.

It should be also be easy to have multiple runners reasonably in sync by carrying around a few simple configuration files, representing the set of images available to the CI.

Ideally, those configuration files could simply be one ansible playbook per chroot. nspawn-runner could have a 'chroot-maintenance' command that runs all the playbooks on their corresponding chroots, and that would be all I need.

ansible and systemd-nspawn

ansible being inadequate as usual, it still does not have a nspawn or machinectl connector, even though the need is there, and one can find requests, pull requests, and implementation attempts by all sorts of people, including me.

However, I don't want to have nspawn-runner depend on random extra plugins. There's a machinectl become plugin available in ansible from buster-backports, but no matter how I read its scant documentation, looked around the internet, and tried all sorts of things, I couldn't manage to figure out what it is for.

This said, simply using systemd-nspawn instead of chroot is quite trivial: use ansible_connection: chroot, set ansible_chroot_exe to this shellscript, and it just works, with things properly mounted, internet access, correct hostnames, and everything:

#!/bin/sh
chroot="$1"
shift
exec systemd-nspawn --console=pipe -qD "$chroot" -- "$@"

I guess that's a, uhm, plan, I guess?

Running playbooks

As an initial prototype, I made nspawn check the list of chroots in /var/lib/nspawn-runner, and for each directory found there, check if there's an equivalent .yaml or .yml file next to nspawn-runner.

For each chroot+playbook combination, it creates an inventory with the right setup, including the custom chroot command, and runs ansible-playbook.

As a prototype it works. I assume once it will see usage there's going to be feedback and fine tuning; meanwhile, I have the beginning of some automated maintenance for the CI chroots.

Next step

It would be nice to also have nspawn-runner create the chroots from configuration files if they are missing, so that a new runner can be deployed with a minimal effort, and it will proceed to generate all the images required in a single command.

For this, I'd like to find a clean way to store the chroot creation command inside the playbooks, to keep just one configuration file per chroot.

I'd also like to have it flexible enough to run debootstrap, as well as commands for different distributions.

Time will tell.

This is probably enough for study/design posts on my blog. Further updates will be in the issue tracker.

pdo debian eng sw systemd

Decameron and content warnings

Giovanni Boccaccio's Decameron is a book in which there's a pandemic, and a group of seven young women and three young men decide to practice social distancing in a secluded villa just outside of Florence.

During two weeks of lockdown, they structure their time wisely, and among other things they spend the evenings telling each other stories. The book is a collection of those stories.

The stories vary from the erotic to the tragic, and may trigger some reader. Boccaccio is aware of that, so he starts each chapter with a one line summary that is explicitly intended as a content warning, to give the reader a choice in whether to read the story, or skip to the next one.

Boccaccio explains this in the book itself:

However, those who read these tales can leave those they dislike and read those they like. I do not want to deceive anybody, and so all these tales bear written at the head a title explaining what they contain.

(link to version in original language)

On top of the very relatable setting, the diversity of stories, the empathy in the narration, the gender inclusivity, and the respect for the reader, it's a pretty good book. It should even be out of copyright!

It was written around year 1350.

pdo life eng

nspawn-runner and btrfs

This post is part of a series about trying to setup a gitlab runner based on systemd-nspawn. I published the polished result as nspawn-runner on GitHub.

systemd-nspawn has an interesting --ephemeral option that sets up temporary copy-on-write filesystem snapshots on filesystems that support it, like btrfs.

Using copy on write means that one could perform maintenance on the source chroots, without disrupting existing CI sessions.

btrfs and copy on write

btrfs snapshots work on subvolumes.

As I understand it, if one uses btrfs subvolume create instead of mkdir, what is inside the resulting directory is managed as a subvolume that can be snapshotted and managed in all sorts of interesting ways.

I managed to delete a subvolume equally well with btrfs subvolume delete and with rm -r.

btrfs subvolume snapshot src dst is similar to cp -a, but it makes a copy-on-write snapshot of a btrfs subvolume.

If I change nspawn-runner to manage each chroot in its own subvolume, I should be able to build on all these features, and systemd-nspawn should be able to do that, too.

There's a cute shortcut to migrate a subdirectory to a subvolume: create the subvolume, then use cp -r --reflink to populate the subvolume with the directory contents.

systemd-nspawn and btrfs

Passing -x/--ephemeral to systemd-nspawn makes it do all the transient copy-on-write work automatically:

# systemd-nspawn -xD buster
Spawning container buster-7fd47ac79296c5d3 on /var/lib/nspawn-runner/t/.#machine.buster0939fbc61fcbca28.
Press ^] three times within 1s to kill container.
root@buster-7fd47ac79296c5d3:~# mkdir foo
root@buster-7fd47ac79296c5d3:~# ls -la
total 12
drwx------ 1 root root  62 Mar 13 16:30 .
drwxr-xr-x 1 root root 154 Mar 13 16:26 ..
-rw------- 1 root root 102 Mar 13 16:26 .bash_history
-rw-r--r-- 1 root root 570 Mar 13 16:26 .bashrc
-rw-r--r-- 1 root root 148 Mar 13 16:26 .profile
drwxr-xr-x 1 root root   0 Mar 13 16:30 foo
root@buster-7fd47ac79296c5d3:~# logout
Container buster-7fd47ac79296c5d3 exited successfully.
root@runner2:/var/lib/nspawn-runner/t# ls -la buster/root/
totale 12
drwx------ 1 root root  56 mar 13 16:26 .
drwxr-xr-x 1 root root 154 mar 13 16:26 ..
-rw------- 1 root root 102 mar 13 16:26 .bash_history
-rw-r--r-- 1 root root 570 mar 13 16:26 .bashrc
-rw-r--r-- 1 root root 148 mar 13 16:26 .profile

It also works on a directory that is not a subvolume, making reflinks of its contents instead of a subvolume snapshot, although this has a performance penalty on setup:

Snapshotting a subvolume:

# time systemd-nspawn -xD buster ls
Spawning container buster-7ab8f4123420b5d5 on /var/lib/nspawn-runner/t/.#machine.bustercd54ef4971229ff5.
Press ^] three times within 1s to kill container.
bin  boot  dev  etc  home  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
Container buster-7ab8f4123420b5d5 exited successfully.

real    0m0,164s
user    0m0,032s
sys 0m0,014s

Reflink-ing a subdirectory:

# time systemd-nspawn -xD buster ls
Spawning container buster-ebc9dc77db0c972d on /var/lib/nspawn-runner/.#machine.buster2ecbcbd1a1a058b8.
Press ^] three times within 1s to kill container.
bin  boot  dev  etc  home  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
Container buster-ebc9dc77db0c972d exited successfully.

real    0m3,022s
user    0m0,326s
sys 0m2,406s

Detecting filesystem type

I can change nspawn-runner to use btrfs-specific features only when /var/lib/nspawn-runner is on btrfs. Here's a command to detect the filesystem type:

# stat -f -c %T /var/lib/nspawn-runner/
btrfs

nspawn-runner updated

I've refactored nspawn-runner splitting backend and frontend code, and implementing multiple backends based on what's the filesystem type of /var/lib/nspawn-runner/.

It works really nicely, and with no special configuration required: if /var/lib/nspawn-runner is on btrfs, things run faster, with less kludges, and one can do maintenance on the base chroots without interfering with running CI jobs.

Next step

The next step is making it easier to configure and maintain chroots. For example, it should be possible to maintain a rolling testing or sid chroot without the need to manually log into it to run apt upgrade.

pdo debian eng sw systemd

Women in science

Timeline of women in science - Wikipedia

this is a timeline of women in science, spanning from ancient history up to the 21st century. While the timeline primarily focuses on women involved with natural sciences such as astronomy, biology, chemistry and physics, it also includes women from the social sciences (e.g. sociology, psychology) and the formal sciences (e.g. mathematics, computer science), as well as notable science educators and medical scientists. The chronological events listed in the timeline relate to both scientific achievements and gender equality within the sciences.

List of women's firsts - Wikipedia

This is a list of women's firsts noting the first time that a woman or women achieved a given historical feat. A shorthand phrase for this development is "breaking the gender barrier" or "breaking the glass ceiling." Other terms related to the glass ceiling can be used for specific fields related to those terms, such as "breaking the brass ceiling" for women in the military and "breaking the stained glass ceiling" for women clergy. Inclusion on the list is reserved for achievements by women that have significant historical impact.

List of Women in Technology International Hall of Fame inductees - Wikipedia

The Women in Technology International Hall of Fame was established in 1996 by Women in Technology International (WITI) to honor women who contribute to the fields of science and technology.

Women in Chemistry – Compound Interest

8 March is International Women’s Day. As in previous years, I’ve put together another edition of this series looking at underappreciated women from chemistry history.

And some men, too:

pdo eng links

Links about privilege

A reality check about the myth of starting from nothing and becoming successful:

Evocative scientific research:

On the other side of privilege:

On reality and representation of reality:

Will they stop at nothing? What are they going to want from us next, our blood? Maybe. How old are you?

pdo eng links

Software development links

Next time we'll iterate on Himblick design and development, Raspberry Pi 4 can now run plain standard Debian, which should make a lot of things easier and cleaner when developing products based on it.

Somewhat related to nspawn-runner, random links somehow related to my feeling that nspawn comes from an ecosystem which gives me a bigger sense of focus on security and solidity than Docker:

I did a lot of work on A38, a Python library to deal with FatturaPA electronic invoicing, and it was a wonderful surprise to see a positive review spontaneously appear! ♥: Fattura elettronica, come visualizzarla con python | TuttoLogico

A beautiful, hands-on explanation of git internals, as a step by step guide to reimplementing your own git: Git Internals - Learn by Building Your Own Git

I recently tried meson and liked it a lot. I then gave unity builds a try, since it supports them out of the box, and found myself with doubts. I found I wasn't alone, and I liked The Evils of Unity Builds as a summary of the situation.

A point of view I liked on technological debt: Technical debt as a lack of understanding

Finally, a classic, and a masterful explanation for a question that keeps popping up: RegEx match open tags except XHTML self-contained tags

pdo eng links devel

Mindustry links

Mindustry is really well made computer game that I enjoyed playing a lot. It is Free Software.

Here are two guides to get a deeper idea about details of the game:

r/Mindustry - How unattended sector defense works (most effective turrets and such) is a useful explanation I found of what happens in Mindustry 6 when I leave a sector to itself.

And #959466 is the RFP (wink, wink!)

pdo eng links

Language links

In English

In Italiano

pdo eng links
Archive of all posts
Archive
Top 15 tags
eng 613 pdo 508 debian 292 sw 222 tips 123 rant 80 life 76 links 74 ita 64 talk 64 songs 61 pub 60 cazzeggio 52 debtags 50 systemd 26
Recent series