The last step of my laptop migration was to fix mime type associations, that seem to associate opening file depending on whatever application was installed last, phases of the moon, and what option is the most annoying.

The state of my system after a fresh install, is that, for application/pdf, xdg-open (used for example by pcmanfm) runs inkscape, and run-mailcap (used for example by neomutt) runs the calibre ebook viewer.

It looks like there are at least two systems to understand, debug and fix, instead of one.

xdg-open

This comes from package xdg-utils, and works using .desktop files:

# This runs inkscape
$ xdg-open file.pdf

There is a tool called xdg-mime that queries what .desktop file is associated with a given mime type:

$ xdg-mime query default application/pdf
inkscape.desktop

You can use xdg-mime default to change an association, and it works nicely:

$ xdg-mime default org.gnome.Evince.desktop application/pdf
$ xdg-mime query default application/pdf
org.gnome.Evince.desktop

However, if you accidentally mistype the name of the .desktop file, it won't complain and it will silently reset the association to the arbitrary default:

$ xdg-mime default org.gnome.Evince.desktop application/pdf
$ xdg-mime query default application/pdf
org.gnome.Evince.desktop
$ xdg-mime default evince.desktop application/pdf
$ echo $?
0
$ xdg-mime query default application/pdf
inkscape.desktop

You can use a GUI like xfce4-mime-settings from the xfce4-settings package to perform the same kind of changes avoiding typing mistakes.

The associations seem to be saved in ~/.config/mimeapps.list

run-mailcap

This comes from the package mime-support

You can test things by running it using --norun:

$ run-mailcap --norun file.pdf
ebook-viewer file.pdf

run-mailcap uses the ~/.mailcap and /etc/mailcap to map mime types to commands. This is what's in the system default:

$ grep application/pdf /etc/mailcap
application/pdf; ebook-viewer %s; test=test -n "$DISPLAY"
application/pdf; calibre %s; test=test -n "$DISPLAY"
application/pdf; gimp-2.10 %s; test=test -n "$DISPLAY"
application/pdf; evince %s; test=test -n "$DISPLAY"

To fix this, I copypasted the evince line into ~/.mailcap, and indeed it gets used:

$ run-mailcap --norun file.pdf
evince file.pdf

There is a /etc/mailcap.order file providing a limited way to order entries in /etc/mailcap, but it can only be manipulated system-wide, and cannot be used for user preferences.

Sadly, this means that if a package changes its mailcap invocation because of, say, a security issue in the former one, the local override will never get fixed.

I am really not comfortable about that. As a workaround, I put this in my ~/.mailcap:

application/pdf; xdg-open %s && sleep 0.3s; test=test -n "$DISPLAY"

The sleep 0.3s is needed because xdg-open exits right after starting the program, and when invoked by mutt it means that mutt could delete the attachment before evince has a chance to open it. I had to use the same workaround for sensible-browser, since the same happens when a browser opens a document in an existing tab. I feel like writing some wrapper about all this that forks the viewer, then waits for an IN_OPEN event on its argument via inotify before exiting.

I wonder if there is any reason run-mailcap could not be implemented as a wrapper to xdg-open.

I reported #964723 elaborating on these thoughts.

My laptop battery started to explode in slow motion. HP requires 10 business days to repair my laptop under warranty, and I cannot afford that length of downtime.

Alternatively, HP quoted me 375€ + VAT for on-site repairs, which I tought was very funny.

For 376.55€ + VAT, which is pretty much exactly the same amount, I bought instead a refurbished ThinkPad X240 with a dual-core I5, 8G of RAM, 250G SSD, and a 1920x1080 IPS display, to use as a spare while my laptop is being repaired. I'd like to thank HP for giving me the opportunity to own a ThinkPad.

Since I'm migrating all my system to the spare and then (hopefully) back, I'm documenting what I need to be fully productive on new hardware.

Install Debian

A basic Debian netinst with no tasks selected is good enough to get going.

Note that if wifi worked in Debian Installer, it doesn't mean that it will work in the minimal system it installed. See here for instructions on quickly bringing up wifi on a newly installed minimal system.

Copy /home

A simple tar of /home is all I needed to copy my data over.

A neat way to do it was connecting the two laptops with an ethernet cable, and using netcat:

# On the source
tar -C / -zcf - home | nc -l -p 12345 -N
# On the target
nc 10.0.0.1 12345 | tar -C / -zxf -

Since the data travel unencrypted in this way, don't do it over wifi.

Install packages

I maintain a few simple local metapackages that depend on the packages I usually used.

I could just install those and let apt bring in their dependencies.

For the build dependencies of the programs I develop, I use mk-build-deps from the devscripts package to create metapackages that make sure they are installed.

Here's an extract from debian/control of the metapackage:

Source: enrico
Section: admin
Priority: optional
Maintainer: Enrico Zini <enrico@debian.org>
Build-Depends: debhelper (>= 11)
Standards-Version: 3.7.2.1

Package: enrico
Section: admin
Architecture: all
Depends:
  mc, mmv, moreutils, powertop, syncmaildir, notmuch,
  ncdu, vcsh, ddate, jq, git-annex, eatmydata,
  vdirsyncer, khal, etckeeper, moc, pwgen
Description: Enrico's working environment

Package: enrico-devel
Section: devel
Architecture: all
Depends:
  git, python3-git, git-svn, gitk, ansible, fabric,
  valgrind, kcachegrind, zeal, meld, d-feet, flake8, mypy, ipython3,
  strace, ltrace
Description: Enrico's development environment

Package: enrico-gui
Section: x11
Architecture: all
Depends:
  xclip, gnome-terminal, qalculate-gtk, liferea, gajim,
  mumble, sm, syncthing, virt-manager
Recommends: k3b
Description: Enrico's GUI environment

Package: enrico-sanity
Section: admin
Architecture: all
Conflicts: libapache2-mod-php, libapache2-mod-php5, php5, php5-cgi, php5-fpm, libapache2-mod-php7.0, php7.0, libphp7.0-embed, libphp-embed, libphp5-embed
Description: Enrico's sanity
 Metapackage with a list of packages that I do not want anywhere near my
 system.

System-wide customizations

I tend to avoid changing system-wide configuration as much as possible, so copying over /home and installing packages takes care of 99% of my needs.

There are a few system-wide tweaks I cannot do without:

• setup postfix to send mail using my mail server
• copy Network Manager system connections files in /etc/NetworkManager/system-connections/
• update-alternatives --config editor

For postfix, I have a little ansible playbook that takes care of it.

Network Manager system connections need to be copied manually: a plain copy and a systemctl restart network-manager are enough. Note that Network Manager will ignore the files unless their owner and permissions are what it expects.

Fine tuning

Comparing the output of dpkg --get-selections between the old and the new system might highlight packages manually installed in a hurry and not added to the metapackages.

Finally, what remains is fixing the sad state of mimetype associations, which seem to associate opening file depending on whatever application was installed last, phases of the moon, and what option is the most annoying.

Currently on my system, PDFs are opened in inkscape by xdg-open and in calibre by run-mailcap. Let's see how long it takes to figure this one out.

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi.

The problem

While testing the cross-compiler, we noticed that the designer library was not being built.

The designer library is needed to build designer plugins, which allow loading, dynamically at runtime, .ui interface files that use custom widgets.

The error the customer got at runtime is: QFormBuilder was unable to create a custom widget of the class '…'; defaulting to base class 'QWidget'.

The library with the custom widget implementation was correctly linked, and indeed the same custom widget was used by the application in other parts of its interface not loaded via .ui files.

It turns out that it is not sufficient, and to load custom widgets automatically, QUiLoader wants to read their metadata from plugin libraries containing objects that implement the QDesignerCustomWidgetInterface interface.

Sadly, building such a library requires using QT += designer, and the designer library, that was not being built by Qt5's build system. This looks very much like a Qt5 bug.

A work around would be to subclass QUiLoader extending createWidget to teach it how to create the custom widgets we need. Unfortunately, the customer has many custom widgets.

The investigation

To find out why designer was not being built, I added -d to the qmake invocation at the end of qtbase/configure, and trawled through the 3.1G build output.

The needle in the haystack seems to be here:

DEBUG 1: /home/build/armhf/qt-everywhere-src-5.15.0/qttools/src/designer/src/src.pro:18: SUBDIRS := uiplugin uitools lib components designer plugins
DEBUG 1: /home/build/armhf/qt-everywhere-src-5.15.0/qttools/src/designer/src/src.pro:23: calling qtNomakeTools(lib components designer plugins)

As far as I can understand, qtNomakeTools seems to be intended to disable building those components if QT_BUILD_PARTS doesn't contain tools. For cross-building, QT_BUILD_PARTS is libs examples, so designer does not get built.

However, designer contains the library part needed for QDesignerCustomWidgetInterface and that really needs to be built. I assume that part should really be built as part of libs, not tools.

The fixes/workarounds

I tried removing designer from the qtNomakeTools invocation at qttools/src/designer/src/src.pro:23, to see if qttools/src/designer/src/designer/ would get built.

It did get built, but then build failed with designer/src/designer and designer/src/uitools both claiming the designer plugin.

I tried editing qttools/src/designer/src/uitools/uitools.pro not to claim the designer plugin when tools is not a build part.

I added the tweaks to the Qt5 build system as debian/patches.

2 hours of build time later...

make check is broken:

make[6]: Leaving directory '/home/build/armhf/qt-everywhere-src-5.15.0/qttools/src/designer/src/uitools'
make[5]: *** No rule to make target 'sub-components-check', needed by 'sub-designer-check'.  Stop.

But since make check doesn't do anything in this build, we can simply override dh_auto_test to skip that step.

Finally, this patch builds a new executable, of an architecture that makes dh_shlibdeps struggle:

dpkg-shlibdeps: error: cannot find library libQt5DesignerComponentssystem.so.5 needed by debian/qtbase5system-armhf-dev/opt/qt5system-armhf/bin/designer (ELF format: 'elf32-little' abi: '0101002800000000'; RPATH: '')
dpkg-shlibdeps: error: cannot find library libQt5Designersystem.so.5 needed by debian/qtbase5system-armhf-dev/opt/qt5system-armhf/bin/designer (ELF format: 'elf32-little' abi: '0101002800000000'; RPATH: '')
…

And we can just skip running dh_shlibdeps on the designer executable.

The result is in the qt5custom git repository.

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi.

These are instructions for building Qt Creator with the custom Qt, so that it can load and use Designer plugins built with it.

Sadly, because of the requirement of being able to load Designer plugins, and because of the requirement of being able to compile custom widgets using the custom Qt and use them in the Designer, we need to also rebuild Qt Creator.

The resulting packaging is at https://github.com/Truelite/qt5custom.

Set up sources

Open the source tarball, and add the Qt Creator packaging:

tar axf qt-creator-enterprise-src-4.12.2.tar.xz
cp -a debian-qtcreator qt-creator-enterprise-src-4.12.2/debian
ln -s qt-creator-enterprise-src-4.12.2.tar.xz qt-creator-enterprise-src_4.12.2.orig.tar.xz

If needed, install the Qt license:

cp qt-license.txt ~/.qt-license

Install build dependencies

You can use apt build-dep to install dependencies manually:

cd qt-creator-enterprise-src-4.12.2
apt build-dep .

Alternatively, you can create an installable .deb metapackage that depends on the build dependencies:

apt install devscripts
mk-build-deps debian-qtcreator/control
apt -f install qt-creator-enterprise-src-build-deps_4.12.2-1_all.deb

Package build

The package is built by debian/rules base on the excellent work done by Debian Qt5 maintainers.

After installing the build dependencies, you can build like this:

cd qt-creator-enterprise-src-4.12.2
debuild -us -uc -rfakeroot

In debian/rules you can configure NUMJOBS with the number of available CPUs in the machine, to have parallel builds.

debian/rules automatically picks qt5custom as the Qt version to use for the build.

NOTE: Qt Creator 4.12.2 will NOT build if qtbase5custom-armhf-dev is installed. One needs to make sure to have qtbase5custom-dev installed, but NOT qtbase5custom-armhf-dev. Despite quite a bit of investigation, I have been unable to understand why, if both are installed, Qt Creator's build chooses the wrong one, and fails the build.

Build output

Building sources generates 4 packages:

• qtcreator-qt5custom: the program
• qtcreator-qt5custom-data: program data
• qtcreator-qt5custom-doc: documentation
• qtcreator-qt5custom-dbgsym: debugging symbols

Using the custom Qt Creator Enterprise

The packages are built with qt5custom and install their content in /opt/qt5custom.

The packages are coinstallable with the version of Qt Creator packaged in Debian.

The custom Qt Creator executable is installed in /opt/qt5custom/bin/qtcreator, which is not in $PATH by default. To run it, you can explitly use /opt/qt5custom/bin/qtcreator. qtcreator ran without an explicit path, runs the standard Debian version.

Installing Designer plugins

Designer plugings can be compiled with qt5custom and installed in /opt/qt5custom/plugins/designer/.

Cross-building with Qt Creator

Once the cross-build Qt5 packages are installed, one should see it appear in the Qt Creator kit configuration, where it can be selected and used normally.

If one sets device type to "Generic Linux Device", chooses a compiler for "arm 32bit" and sets Qt Version to "qt5custom-armhf", one can smoothly cross-compile and execute and debug the built program directly on the device.

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi.

I split building Qt5 for armhf development in two parts: one cross-build environment to be installed on amd64 develpment systems, and a runtime part to be installed on the target armhf hardware.

Building a Qt5 cross-building environment builds a mix of armhf and amd64 binares: the amd64 tools to use for cross-building, like moc, qmake, plugins for Qt Creator, and so on; armhf headers and libraries to use at cross-build time; armhf shared libraries to use at runtime.

The procedure I came up with builds a devel package for amd64 development machines, which contains everything, and a second package that extracts from it only what is needed at runtime.

The cross-build environment is coinstallable both with the version of Qt distributed with Debian, and with the amd64 custom Qt development package.

The current build is sadly using -skip qtwebengine, because I have had no success so far getting QtWebEngine to compile as part of a cross-build Qt setup (the last road bump I can't overcome is nss and nspr not being coinstallable on amd64 and armhf, while both seem to be needed for it).

The resulting packaging is at https://github.com/Truelite/qt5custom.

Set up sources

Open the source tarball, and add the amd64 packaging:

tar axf qt-everywhere-src-5.15.0.tar.xz
cp -a debian-cross qt-everywhere-src-5.15.0/debian

If needed, install the Qt license:

cp qt-license.txt ~/.qt-license

If debugging information are not needed in armhf development, remove --no-strip from ./configure invocation in the rules file, to build significantly smaller .deb packages.

Install build dependencies

Install cross-compilers:

dpkg --add-architecture armhf
apt install crossbuild-essential-armhf

You can use apt build-dep to install dependencies manually:

cd qt-everywhere-src-5.15.0
apt build-dep .
apt -a armhf build-dep .

Alternatively, you can create installable .deb metapackages that depends on the build dependencies:

apt install devscripts
mk-build-deps --host-arch amd64 debian-cross/control
mk-build-deps --host-arch armhf debian-cross/control
apt -f install qt-everywhere-cross-build-deps_5.15.0-1_amd64.deb qt-everywhere-cross-cross-build-deps_5.15.0-1_armhf.deb

Note that there are two sets of dependencies: one of amd64 packages, and one of armhf packages.

Building the cross-build environment

After installing the build dependencies, you can build like this:

cd qt-everywhere-src-5.15.0
fakeroot debian/rules binary

In debian/rules you can configure NUMJOBS with the number of available CPUs in the machine, to have parallel builds.

This will build a package with the cross-build development environment for amd64, called qtbase5custom-armhf-dev

Building the runtime environment

To generate the runtime package for armhf, one needs to have the cross-build package (qtbase5custom-armhf-dev) installed in the system together with its build dependencies.

At that point, the armhf runtime package can be built using the debian-armhf directory without further sources:

apt install crossbuild-essential-armhf debhelper qtbase5custom-armhf-dev*_amd64.deb qt-everywhere-src-cross-build-deps*_armhf.deb
mkdir runtime
cp -a debian-armhf runtime/debian
cd runtime
dpkg-buildpackage -a armhf

Building the runtime environment generates:

• a libqt5custom package for armhf, installable on the target devices, containing the runtime Qt libraries and depending on the packages that they need to run;
• a libqt5custom-dbgym package for armhf with debugging symbols, to use for debugging on the target hardware.

If, while generating the cross-build environment, --no-strip was removed, the libqtcustom-dbgsym package with debugging symbols will not be generated.

Using the cross-build environment

These install their content in /opt, and are coninstallable with the version of Qt distributed in Debian, and with the custom Qt packages for amd64.

One needs to be careful not to create programs that link, either directly or indirectly, with more than one of these coinstalled Qt, because the in memory layout of objects could be different and incompatible, causing unexpected results.

Selecting which Qt version to use: qtchooser

These Qt custom packages integrate with qtchooser to select the version of Qt to use at compile time.

qtchooser --list-versions lists available versions. One can choose what to use by exporting QT_SELECT:

# apt install qtchooser qt5-qmake qt5-default
$ qtchooser --list-versions
4
5
qt4-x86_64-linux-gnu
qt4
qt5-x86_64-linux-gnu
qt5
qt5custom-x86_64-linux-gnu
qt5custom
qt5custom-armhf-x86_64-linux-gnu
qt5custom-armhf

$ qmake --version
QMake version 3.1
Using Qt version 5.11.3 in /usr/lib/x86_64-linux-gnu

$ export QT_SELECT=qt5custom-armhf
$ qmake --version
QMake version 3.1
Using Qt version 5.15.0 in /opt/qt5custom-armhf/lib

Cross-building software using custom Qt

One just needs to export QT_SELECT=qt5custom-armhf in the environment, then proceed to build normally:

export QT_SELECT=qt5custom-armhf
fakeroot ./debian/rules clean binary

Or:

export QT_SELECT=qt5custom-armhf
qmake file.pro

If switching from one Qt to another, it is possible that the makefiles created by one qmake are not working well with the other. In that case, one can just remove them and regenerate them.

The build result is ready to be copied into, and run in, the target armhf device.

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi.

A customer needs a procedure for a custom build of Qt5 5.15, the last LTS release of Qt 5.

They develop for industrial systems that are managed by an amd64 industrial computer. This computer is accessed either through an attached panel touch screen, or through touch screens driven by Raspberry Pi clients connected via an internal ethernet network.

The control interfaces use mostly a full screen Qt5 application. The customer relies heavily on Qt5, has a full Enterprise license, and needs to stay on top of the most recent releases, to make use of new features or bug fixes that have made it upstream since the last Debian stable was released.

This is a list of requirements for this job:

• Build .deb packages of the custom builds of Qt5, so they can be integrated with the existing provisioning infrastructure
• Easily repackage hopefully at least new Qt minor versions
• Custom builds should be coinstallable with the standard Debian Qt5 packages, to be able to use existing Qt-based Debian packages without rebuilding them
• One needs to be able to develop custom widgets, and use them in the Form Editor in Qt Creator
• One needs to be able to load custom widgets via .ui files at runtime
• One needs to develop amd64 Qt5 applications
• One needs to develop armhf Qt5 applications
• One needs to develop armhf Qt5 applications from Qt Creator, with the nice feature it has to cross-compile them on the fast amd64 development machine, and run and debug them directly on a network-connected device
• One needs to package the resulting amd64 or armhf binaries in .deb format, so they can be integrated with the existing provisioning infrastructure.

To make things easier, .deb packages are for internal use only and do not need to be compliant with Debian policy.

I estimate a difficulty level of: "Bring the One Ring to Mount Doom and remember to get milk on the way back".

The journey begins.

The resulting packaging is at https://github.com/Truelite/qt5custom.

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi.

First step, build Qt5 5.15 packages for amd64.

To prevent conflicting with Debian Qt packages, we'll install everything in /opt.

We can install qtchooser configuration files to allow developers to easily switch between Debian's standard Qt version or the custom version, at will.

The resulting packaging is at https://github.com/Truelite/qt5custom.

Set up sources

Open the source tarball, and add the amd64 packaging:

tar axf qt-everywhere-src-5.15.0.tar.xz
cp -a debian-amd64 qt-everywhere-src-5.15.0/debian

If needed, install the Qt license:

cp qt-license.txt ~/.qt-license

Install build dependencies

You can use apt build-dep to install dependencies manually:

cd qt-everywhere-src-5.15.0
apt build-dep .

Alternatively, you can create an installable .deb metapackage that depends on the build dependencies:

apt install devscripts
mk-build-deps debian-amd64/control
apt -f install qt-everywhere-src-build-deps_5.15.0-1_amd64.deb

Package build

The package is built by debian/rules base on the excellent work done by Debian Qt5 maintainers

After installing the build dependencies, you can build like this:

cd qt-everywhere-src-5.15.0
fakeroot debian/rules binary

In debian/rules you can configure NUMJOBS with the number of available CPUs in the machine, to have parallel builds.

Build output

Building sources generates 4 packages:

• libqt5custom: the runtime environment
• libqt5custom-dbgsym: debugging symbols for the runtime environment
• qtbase5custom-dev: the build environment
• qtbase5custom-dev-dbgsym: debugging symbols for the build environment

qtbase5custom-dev and libqt5custom are needed for development; only libqt5custom is needed to run build programs.

Using custom Qt for amd64

These Qt custom packages install their content in /opt, and are coninstallable with the version of Qt distributed in Debian.

One needs to be careful not to create programs that link, either directly or indirectly, with both the Debian Qt and the custom Qt, because the in memory layout of objects could be different and incompatible, causing unexpected results.

Selecting which Qt version to use: qtchooser

These Qt custom packages integrate with qtchooser to select the version of Qt to use at compile time.

qtchooser --list-versions lists available versions. One can choose what to use by exporting QT_SELECT:

# apt install qtchooser qt5-qmake qt5-default
$ qtchooser --list-versions
4
5
qt4-x86_64-linux-gnu
qt4
qt5-x86_64-linux-gnu
qt5
qt5custom-x86_64-linux-gnu
qt5custom

$ qmake --version
QMake version 3.1
Using Qt version 5.11.3 in /usr/lib/x86_64-linux-gnu

$ export QT_SELECT=qt5custom
$ qmake --version
QMake version 3.1
Using Qt version 5.15.0 in /opt/qt5custom/lib

Building software using custom Qt

One just needs to export QT_SELECT=qt5custom in the environment, then proceed to build normally:

export QT_SELECT=qt5custom
fakeroot ./debian/rules clean binary

Or:

export QT_SELECT=qt5custom
qmake file.pro

If switching from one Qt to another, it is possible that the makefiles created by one qmake are not working well with the other. In that case, one can just remove them and regenerate them.

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

One day after the first deploy, we went to check how the system was doing, and noticed some fine tuning to do, some pretty much urgent.

(continue reading)

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

Another surprise hits us at the last moment: if the system boots without an HDMI monitor plugged in, no framebuffer device is ever created, and X will not start, lightdm will give up after some tries, and even if one plugs in a monitor afterwards, it will stay blank until a reboot or some kind of manual intervention.

As a workaround, one can configure the bootloader to force a specific HDMI configuration. This post documents how we did it.

(continue reading)

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

Provisioning a SD card starting from the official raspbian-lite is getting quite slow, since there are a lot of packages to install.

It would be significantly faster if we could take a SD card, partition it from scratch, then untar the boot and rootfs partition contents into them.

Here's how.

(continue reading)