Last 10 blog posts

Inspired by this post about documentation¹ I started a systemd/documentation page in the Debian wiki.

Systemd has an excellent reference documentation in its manpages, but it does a lot of things, and a reference documentation isn't the best starting point for getting introduced to them.

I would like to see a bit more documentation of the kind that sits between a systemctl start|stop|status and the reference manpages. Things like simple HOWTO posts on how to get a simple job done, or high-level explanations of how some specific feature works.

I put some of what I know and used (or wrote) into systemd/documentation, I'll try to add to it when I find more, and I encourage you to do the same.

If you remember an article that has been useful to you and it is missing from the page, link it.

If you found a systemd feature useful to get a task done, write a little HOWTO to the wiki or your blog, and link it.

If you don't use systemd, go ahead and start a similar index for the init system that you use, and link it, too.

I'd love it if each init system we have in Debian had an excellent documentation index in the wiki!

There is a quote that goes around allegedly attributed to Albert Einstein:

Let’s not pretend that things will change if we keep doing the same things. A crisis can be a real blessing to any person, to any nation. For all crises bring progress.

Creativity is born from anguish, just like the day is born form the dark night. It’s in crisis that inventiveness is born, as well as discoveries made and big strategies. He who overcomes crisis, overcomes himself, without getting overcome. He who blames his failure to a crisis neglects his own talent and is more interested in problems than in solutions. Incompetence is the true crisis. The greatest inconvenience of people and nations is the laziness with which they attempt to find the solutions to their problems.

There’s no challenge without a crisis. Without challenges, life becomes a routine, a slow agony.

There’s no merit without crisis. It’s in the crisis where we can show the very best in us. Without a crisis, any wind becomes a tender touch. To speak about a crisis is to promote it. Not to speak about it is to exalt conformism. Let us work hard instead. Let us stop, once and for all, the menacing crisis that represents the tragedy of not being willing to overcome it.

I like the idea of considering more valuable how a group overcomes a crisis, rather than how a group avoids it.

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

We've started implementing reloading of the media player when media on disk changes.

One challenge when doing that, is that libreoffice doesn't always stop. Try this and you will see that the presentation keeps going:

$ loimpress --nodefault --norestore --nologo --nolockcheck --show example.odp
$ pkill -TERM loimpress

It turns out that loimpress forks various processes. After killing it, these processes will still be running:

/usr/lib/libreoffice/program/oosplash --impress --nodefault --norestore --nologo --nolockcheck --show talk.odp
/usr/lib/libreoffice/program/soffice.bin --impress --nodefault --norestore --nologo --nolockcheck --show talk.odp

Is there a way to run the media players in such a way that, if needed, they can easily be killed, together with any other process they might have spawned meanwhile?

systemd-run

Yes there is: systemd provides a systemd-run command to run simple commands under systemd's supervision:

$ systemd-run --scope --slice=player --user \
      loimpress --nodefault --norestore --nologo --nolockcheck --show media/talk.odp

This will run the player contained in a cgroup with a custom name, and we can simply use that name to stop all the things:

$ systemctl --user stop player.slice

Resulting python code

The result is this patch which simplifies the code, and isolates and easily kills all subprocesses run as players.

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

Another nice to have in a system like Himblick is the root filesystem mounted readonly, with a volatile tempfs overlay on top. This would kind of always guarantee a clean boot without leftovers from a previous run, especially in a system where the most likely mode of shutdown is going to be pulling the plug.

This won't be a guarantee about SD issues developing over time in such a scenario, but it should at least cover the software side of things.

In theory, systemd supports this out of the box with the systemd.volatile=overlay kernel command line option, including integrating this with journald and the way other things get mounted.

In practice:

• this feature is available starting from systemd 242 (Debian Buster has 241). Systemd 242 is however available from backports.
• Debian's default initrd, initramfs-tools, does not run systemd.
• dracut does, but Debian's dracut does not yet know how to include the relevant parts of systemd
• Raspbian doesn't boot with an initrd by default, althoug it can.

While things catch up, dracut has a rootovl option that implements something similar.

The procedure becomes, roughly:

# apt install --no-install-recommends dracut
# echo filesystems+=overlay > /etc/dracut.conf.d/overlay.conf
# dracut /boot/initrd.img
# sed -ri -e '/$/ rootovl/' /boot/cmdline.txt" rootovl" in cmdline
# echo "initramfs initrd.img" >> /boot/config.txt

Here's how it ended up in python:

    def setup_readonly_root(self):
        """
        Setup a readonly root with a tempfs overlay
        """
        # Set up a readonly root using dracut's 'rootovl' feature.
        # Eventually do this with systemd's systemd.volatile=overlay option.
        # See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945596
        self.apt_install("dracut")
        self.write_file("/etc/dracut.conf.d/overlay.conf", "filesystems+=overlay\n")
        self.run(["dracut", "--force", "/boot/initrd.img", "4.19.75-v7l+"])
        with self.edit_kernel_commandline("/boot/cmdline.txt") as parts:
            # Add 'rootovl' to /etc/cmdline
            if "rootovl" not in parts:
                parts.append("rootovl")

        # Add initramfs initrd.img to config.txt
        with self.edit_text_file("/boot/config.txt") as lines:
            if "initramfs initrd.img" not in lines:
                lines.append("initramfs initrd.img")

Packages worth removing

• rsyslog is not needed in this setup. It can just be removed, as journald works well with volatile systems
• dphys-swapfile should be removed, in case it gets activated and tries to expand the ram by creating a swapfile in ram
• fake-hwclock is not needed since we disabled filesystem checks

Other links

Here is a series of link to other resources for doing something similar:

• https://medium.com/@andreas.schallwig/how-to-make-your-raspberry-pi-file-system-read-only-raspbian-stretch-80c0f7be7353
• http://hallard.me/raspberry-pi-read-only/
• https://wiki.debian.org/ReadonlyRoot
• https://learn.adafruit.com/read-only-raspberry-pi/overview
• https://wiki.archlinux.org/index.php/Overlay_filesystem
• https://www.domoticz.com/wiki/Setting_up_overlayFS_on_Raspberry_Pi

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

After seeing lots of automatic mount/umount notifications during provisioning, we wondered if it would be possibile to temporarily disable them while we're working on the SD card.

It turns out that it's possible, and here's a convenient python context manager to do it cleanly, based on /usr/lib/udisks2/udisks2-inhibit, but adding the possibility of inhibiting automounting only on one specific device:

    @contextmanager
    def pause_automounting(self, dev: Dict[str, Any]):
        """
        Pause automounting on the device for the duration of this context
        manager
        """
        # See /usr/lib/udisks2/udisks2-inhibit
        devpath = dev["path"]
        rules_dir = "/run/udev/rules.d"
        os.makedirs(rules_dir, exist_ok=True)
        rule_file = os.path.join(rules_dir, "90-udisks-inhibit-" + devpath.replace("/", "_") + ".rules")
        with open(rule_file, "wt") as fd:
            print('SUBSYSTEM=="block", ENV{DEVNAME}=="' + devpath + '*", ENV{UDISKS_IGNORE}="1"', file=fd)
            fd.flush()
            os.fsync(fd.fileno())
        run(["udevadm", "control", "--reload"])
        run(["udevadm", "trigger", "--settle", "--subsystem-match=block"])
        try:
            yield
        finally:
            os.unlink(rule_file)
            run(["udevadm", "control", "--reload"])
            run(["udevadm", "trigger", "--settle", "--subsystem-match=block"])

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

Finally, we have enough pieces to start working on the media player. It's been way more work that expected getting to this point, and I hope that this series of posts could help others getting a faster start.

To begin with, we'd like to be able to show:

• PDF files automatically looping through the pages
• Image galleries automatically looping
• Looping videos
• ODP presentations

Configuring the screen

The first thing to do on startup is to configure the screen based on the himblick.conf settings:

    def configure_screen(self):
        """
        Configure the screen based on himblick.conf
        """
        # Set screen orientation
        orientation = self.settings.general("screen orientation")
        if orientation:
            run(["xrandr", "--orientation", orientation])

        mode = self.settings.general("screen mode")
        if mode:
            res = run(["xrandr", "--query"], capture_output=True, text=True)
            re_output = re.compile(r"^(\S+) connected ")
            for line in res.stdout.splitlines():
                mo = re_output.match(line)
                if mo:
                    output_name = mo.group(1)
                    break
            else:
                output_name = None
            run(["xrandr", "--output", output_name, "--mode", mode])

This had the extra complication of needing to parse xrandr --query output to figure out the name of the HDMI output in use, since the RaspberryPi 4 has two of them. It would be nice if xrandr could join the ranks of tools with a machine parsable output, like for example lsblk is doing.

Scanning the media directory

The next step is finding what to play. We scan the media directory looking at file mimetypes, to avoid having to hardcode all the possible file extension that image and video files can have. Then we group media by type, and pick the group with the most recent files:

    def find_presentation(self, path):
        """
        Find the presentation to play from a given media directory
        """
        if not os.path.isdir(path):
            return None
        pdf = PDFPresentation()
        videos = VideoPresentation()
        images = ImagePresentation()
        odp = ODPPresentation()
        all_players = [pdf, videos, images, odp]

        for fn in os.listdir(path):
            abspath = os.path.abspath(os.path.join(path, fn))
            base, ext = os.path.splitext(fn)
            mimetype = mimetypes.types_map.get(ext)
            if mimetype is None:
                log.info("%s: mime type unknown", fn)
                continue
            else:
                log.info("%s: mime type %s", fn, mimetype)
            if mimetype == "application/pdf":
                pdf.add(abspath)
            elif mimetype.startswith("image/"):
                images.add(abspath)
            elif mimetype.startswith("video/"):
                videos.add(abspath)
            elif mimetype == "application/vnd.oasis.opendocument.presentation":
                odp.add(abspath)

        player = max(all_players, key=lambda x: x.mtime)
        if not player:
            return None
        return player

Caffeinate

We don't want power management to kick in and turn our signage screens black, so we are running all the media players under caffeine.

We automated with a simple subprocess.run wrapper. Note the -- to prevent caffeinate from choking on the options passed to the actual media players:

def run(cmd: List[str], check: bool = True, **kw) -> subprocess.CompletedProcess:
    """
    Logging wrapper to subprocess.run.

    Also, default check to True.
    """
    log.info("Run %s", " ".join(shlex.quote(x) for x in cmd))
    return subprocess.run(cmd, check=check, **kw)


class Presentation:
    """
    Base class for all presentation types
    """
    def run_player(self, cmd, **kw):
        """
        Run a media player command line, performing other common actions if
        needed
        """
        # Run things under caffeinate
        # See also: https://stackoverflow.com/questions/10885337/inhibit-screensaver-with-python
        cmd = ["caffeinate", "--"] + cmd
        run(cmd, **kw)

Showing PDFs

okular seems to be the only PDF reader in Debian that can be convinced to do looping non interactive full screen presentations, with only a bit of tampering with its configuration files:

class PDFPresentation(SingleFileMixin, Presentation):
    def run(self):
        log.info("%s: PDF presentation", self.fname)

        confdir = os.path.expanduser("~/.config")
        os.makedirs(confdir, exist_ok=True)

        # TODO: configure slide advance time

        # Configure okular
        with open(os.path.expanduser(os.path.join(confdir, "okularpartrc")), "wt") as fd:
            print("[Core Presentation]", file=fd)
            print("SlidesAdvance=true", file=fd)
            print("SlidesAdvanceTime=2", file=fd)
            print("SlidesLoop=true", file=fd)
            print("[Dlg Presentation]", file=fd)
            print("SlidesShowProgress=false", file=fd)
            # print("SlidesTransition=GlitterRight", file=fd)

    # Silence a too-helpful first-time-run informational message
        with open(os.path.expanduser(os.path.join(confdir, "okular.kmessagebox")), "wt") as fd:
            print("[General]", file=fd)
            print("presentationInfo=4", file=fd)

        # Remove state of previous okular runs, so presentations begin at the
        # beginning
        docdata = os.path.expanduser("~/.local/share/okular/docdata/")
        if os.path.isdir(docdata):
            shutil.rmtree(docdata)

        self.run_player(["okular", "--presentation", "--", self.fname])

I was surprised at how looping a PDF presentation doesn't seem to be a well supported use case in PDF viewers. If it's somewhat painful to do it in okular, it's downright impossible to do it with evince: try evince --fullscreen --presentation: slides won't advance, and it still shows a toolbar!

Showing images

class ImagePresentation(FileGroupMixin, Presentation):
    def run(self):
        self.files.sort()
        log.info("Image presentation of %d images", len(self.files))
        with tempfile.NamedTemporaryFile("wt") as tf:
            for fname in self.files:
                print(fname, file=tf)
            tf.flush()

            # TODO: adjust slide advance time
        self.run_player(["feh", "--filelist", tf.name, "--fullscreen",
                         "--hide-pointer", "--slideshow-delay", "1.5"])

feh does everything needed and more. It seems to support our use case explictly, with useful knobs exposed on the command line, clean, straightforward, beautiful!

Showing videos

Most internet posts about playing media on Raspberry Pi, suggest omxplayer. After trying it it looked quite worrysome, as it seemed to fail with any media format not supported in hardware, there did not seem to be a way to ask it whether a file would be in a playable format or not, and one of the failures left the screen in the wrong resolution.

We would like the media player to be able to play the widest possible range of media, hardware accelerated if possible, software if not.

Luckily, it turned out that vlc can use the Raspberry Pi 4 hardware acceleration, and playing a 1920x1080 video full screen on it would consume only 4% of CPU, which is the same that omxplayer was using.

That was very relieving, as vlc can also play a wide range of media, has excellent support for gapless looping, can be invoked without a UI, and can even do playlists of multiple media.

Here is the corresponding player code:

class VideoPresentation(FileGroupMixin, Presentation):
    def run(self):
        self.files.sort()
        log.info("Video presentation of %d videos", len(self.files))
        with tempfile.NamedTemporaryFile("wt", suffix=".vlc") as tf:
            for fname in self.files:
                print(fname, file=tf)
            tf.flush()

            self.run_player(
                    ["cvlc", "--no-audio", "--loop", "--fullscreen",
                        "--video-on-top", "--no-video-title-show", tf.name])

Showing presentations

The code here is quite straightforward, but it took a while to put together that command line:

class ODPPresentation(SingleFileMixin, Presentation):
    def run(self):
        log.info("%s: ODP presentation", self.fname)
        self.run_player(["loimpress", "--nodefault", "--norestore", "--nologo", "--nolockcheck", "--show", self.fname])

I was surprised that I could not find a way to tell Impress to just play a presentation without other things getting in the way. Even like that, there is a moment in which the UI can be seen to come up on the screen before being covered by the full screen presentation.

There is also no way to force a presentation to loop or to advance slides after a given timeout: both features need to be set in the presentation itself.

People will have to do a test run of their presentations with a checklist before putting them on the player. It would have been nice to have an easy way to guarantee that a presentation wouldn't get stuck on the player.

That is not a requirement for now anyway. If it ever becomes one, I guess we can always write code to check and tweak the .odp presentation file: it's thankfully a well known and open format.

Auditing recommends

So far we installed everything with --no-install-recommends, but it's risky to do so when dealing with packages with many dependencies like vlc, okular, and impress.

Aptitude offers the possibility to audit recommends: Views / Audit Recommendations will show a list of recommended but not installed packages.

That turned out some font packages that it's maybe nice to have, and libreoffice-avmedia-backend-vlc that may come in handy if people decide to play presentations with embedded videos.

Existing presentation software

Our needs for media playing so far have been simple. Should they become more complex, here are some pointers to existing, more featureful projects:

• Pi Presents
• Info Beamer
• ffdiaporama

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

When powered on, the Pi units should go straight into X, and start the media player.

X autologin has long been a gripe of mine, and surprisingly hard to do right and reliably, integrating well with PAM, with no greeters or switches to text mode flashing quickly on the screen, and so on.

I once wrote nodm. Then lightdm-autologin-greeter, so I didn't have to maintain a whole display manager for this. Lightdm's autologin used to be insufficient, because when the X session ended, lightdm assumed one wanted to log out and showed the greeter again.

Now I'm very pleased to see that, in 2019, almost 2020, just setting autologin-user in lightdm does the right thing, and I'm very happy to be able to remove some of the kludges from the geological strata of kludges that accreted over the years to work around these kinds of things.

First thing first, do not wait for network to be online during boot. We don't need it in a media player that should be able to work just as well offline:

       # Do not wait for being online to finish boot
       chroot.systemctl_disable("systemd-networkd-wait-online.service", mask=True)

Setting up autologin now really is quite beautifully straightforward:

 - name: "Install X dependencies"
   apt:
      pkg:
      # basic X server
       - xserver-xorg
       - lightdm
      state: present
      update_cache: no
      install_recommends: no

 - name: "Enable lightdm autologin"
   lineinfile:
      path: /etc/lightdm/lightdm.conf
      regexp: ".*autologin-user\\s*=\\s*($|pi)"
      line: "autologin-user = pi"

A nice trick for a media display: X without a mouse cursor accidentally hovering like a fly over things:

 - name: "Disable mouse cursor in X"
   lineinfile:
      path: /etc/lightdm/lightdm.conf
      regexp: "(^|#)xserver-command\\s*=\\s*"
      line: "xserver-command = X -nocursor"

Finally, we need to start the player in the X session.

There are quite a lot of ways to autostart things in X sessions, and the Xsession page in the Debian Wiki has excellent documentation on it.

Since the machine will only be a media player, as a first attempt we decided to try and be the X session, starting the player directly with no desktop environment, no window manager, nothing, ohne alles:

 - name: "Use himblick player as X session"
   copy:
      dest: /home/pi/.xsession
      owner: pi
      group: pi
      mode: '0644'
      content: |
          exec /usr/bin/himblick player

This way, the player can stay there and do its thing without anything popping up in its way, and if it dies, X ends, and lightdm restarts it all.

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

Time to setup ssh. We want to have admin access to the pi user, and we'd like to have a broader access to a different, locked down user, to use to manage media on the boxes via sftp.

The first step is to mount the exFAT media partition into /srv/media:

---
 - name: "Install exfat drivers"
   apt:
      name: exfat-fuse,exfat-utils
      state: present
      update_cache: no

 - name: "Create /srv directory"
   file:
      path: "/srv"
      state: directory
      owner: root
      group: root
      mode: 0755

 - name: "Create /media mount point"
   file:
      path: "/srv/media"
      state: directory
      owner: pi
      group: pi
      mode: 0755

 - name: "Configure mounting media directory"
   copy:
      src: srv-media.mount
      dest: /etc/systemd/system/srv-media.mount
      owner: root
      group: root
      mode: 0644

Mounting exFAT before Linux kernel 5.4 requires FUSE. Using a mount unit allows us to bring up the mount after FUSE is up, and get it mounted at boot reliably.

We add a round of filesystem checking, too: if people plug the SD into a computer to load media into it, we can't be sure that they unmount it cleanly.

This is srv-media.mount; note that .mount unit names need to match the path of the mount point:

[Unit]
Description=Media Directory (/srv/media)
Before=local-fs.target
After=sys-fs-fuse-connections.mount

[Mount]
What=/dev/disk/by-label/media
Where=/srv/media
Type=exfat
Options=uid=0,gid=1001,fmask=117,dmask=007,rw,noatime,nosuid,nodev
ExecStartPre=-/sbin/fsck.exfat -a /dev/disk/by-label/media

[Install]
WantedBy=local-fs.target

gid 1001 is the media group id, shared by the pi user that runs the media player, and by the media user that does sftp. We make everything the media mount group-writable by the media user so both users can access it.

Next, we prepare a chroot jail for the media user. The root of the jail needs to be writable only by root, so we bind mount the media directory inside it:

 - name: "Create the chroot jail for media: /srv"
   file:
      path: "/srv"
      state: directory
      owner: root
      group: root
      mode: 0755

 - name: "Create the chroot jail for media: /srv/jail"
   file:
      path: "/srv/jail"
      state: directory
      owner: root
      group: root
      mode: 0755

 - name: "Create the chroot jail for media: /srv/jail/media"
   file:
      path: "/srv/jail/media"
      state: directory
      owner: root
      group: media
      mode: 0755

 - name: "Bind mount /srv/media under /srv/jail/media"
   copy:
      src: srv-jail-media.mount
      dest: /etc/systemd/system/srv-jail-media.mount
      owner: root
      group: root
      mode: 0644

This is the srv-jail-media.mount mount unit, neatly ordered to start after /srv/media is mounted:

[Unit]
Description=Media Directory in sftp jail (/srv/jail/media)
Before=local-fs.target
After=srv-media.target

[Mount]
What=/srv/media
Where=/srv/jail/media
Type=none
Options=bind

[Install]
WantedBy=local-fs.target

Finally, the ssh configuration:

---
 - name: "Disable ssh password authentication"
   lineinfile:
      path: /etc/ssh/sshd_config
      regexp: '\bPasswordAuthentication\b'
      line: 'PasswordAuthentication no'

 - name: "Install ssh admin access key"
   authorized_key:
      user: pi
      state: present
      key: "{{SSH_AUTHORIZED_KEY}}"
   when: SSH_AUTHORIZED_KEY is defined

 - name: "Install ssh media access key"
   authorized_key:
      user: media
      state: present
      key: "{{SSH_MEDIA_PUBLIC_KEY}}"
   when: SSH_MEDIA_PUBLIC_KEY is defined

 - name: "Install media access key for the pi user"
   copy:
      dest: "/home/pi/.ssh/id_media"
      content: "{{SSH_MEDIA_PRIVATE_KEY}}"
      owner: pi
      group: pi
      mode: 0600
   when: SSH_MEDIA_PRIVATE_KEY is defined

 - name: "Configure internal sftp, so ssh does not need binaries inside the jail"
   lineinfile:
      path: /etc/ssh/sshd_config
      regexp: ".*Subsystem\\s+sftp"
      line: "Subsystem sftp internal-sftp"

 - name: "Configure sftp chroot jail for user media"
   blockinfile:
      path: /etc/ssh/sshd_config
      block: |
         Match User media
              ChrootDirectory /srv/jail
              AllowTcpForwarding no
              X11Forwarding no
              ForceCommand internal-sftp

Don't forget to enable the media units:

       # Enable the /srv/media mount point, which ansible, as we run it
       # now, is unable to do
       chroot.systemctl_enable("srv-media.mount")
       chroot.systemctl_enable("srv-jail-media.mount")