Tag sw

Latest posts for tag sw

2016-12-25 13:38:33+01:00

"Intervallo RAI" generator

During holiday idling, I made a thing to generate picture slideshows similar to RAI's iconic "Intervallo"

You can get it at https://github.com/spanezz/intervallo

Usage:

$ intervallo --help
usage: intervallo [-h] [--font file.ttf] [--audio file.mp3] [--duration sec]
                  imgfile [imgfile ...]

Create an Intervallo RAI out of a collection of images.

positional arguments:
  imgfile           input image files

optional arguments:
  -h, --help        show this help message and exit
  --font file.ttf   Font to use for subtitles
  --audio file.mp3  Audio track
  --duration sec    Time for each image in seconds

For example:

./intervallo --font DejaVuSerif.ttf --audio Paradisi-Toccata.mp3 *.jpg

The images are captioned with their file name, without extension. You may want to rename the image files to have nice descriptive names.

For some audio to use, you can try https://archive.org/details/IntervalloRai-Paradisi

Example

https://www.youtube.com/watch?v=ZCERwjTB4ck

debian eng pdo sw
2016-11-09 10:10:42+01:00

On SPF

I woke up this morning with some Django server error mails in my inbox:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 9: ordinal not in range(128)
…
 'REMOTE_USER': u'…-guest@users.alioth.debian.org',

I did what one does in cases like these, I tried to fix the bug and mailed …-guest@users.alioth.debian.org asking them to try again and let me know if it works.

I get a bounce:

  <Actual user's email>
    (generated from …-guest@users.alioth.debian.org)
    SMTP error from remote mail server after MAIL FROM:<enrico@enricozini.org> SIZE=3948:
    host … […]: 550 Please see http://www.openspf.org/Why?id=enrico%40enricozini.org&ip=2001%3a41c8%3a1000%3a21%3a%3a21%3a21&receiver=bq :
    Reason: mechanism

I resent the mail to the actual user's address, and it went through. Job done, at least until they get back to me telling me that my fix didn't work.

Lessons learnt:

debian eng pdo sw
2016-11-02 16:54:06+01:00

schroot connector for ansible

I accidentally made an ansible connector plugin for schroot. I haven't even used ansible yet, so I have no idea what I am doing.

You can choose the chroot type by setting something like schroot_type: jessie in your variables.

You can install this locally as connection_plugins/schroot.py.

# Based on chroot.py (c) 2013, Maykel Moya <mmoya@speedyrails.com>
# Based on chroot.py (c) 2015, Toshio Kuratomi <tkuratomi@ansible.com>
# (c) 2016, Enrico Zini <enrico@debian.org>
#
# This is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Ansible is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

import distutils.spawn
import os
import os.path
import pipes
import subprocess
import traceback

from ansible import constants as C
from ansible.errors import AnsibleError
from ansible.plugins.connection import ConnectionBase, BUFSIZE
from ansible.module_utils.basic import is_executable
from ansible.utils.unicode import to_bytes

try:
    from __main__ import display
except ImportError:
    from ansible.utils.display import Display
    display = Display()


class Connection(ConnectionBase):
    ''' Local chroot based connections '''

    transport = 'schroot'
    has_pipelining = True
    # su currently has an undiagnosed issue with calculating the file
    # checksums (so copy, for instance, doesn't work right)
    # Have to look into that before re-enabling this
    become_methods = frozenset(C.BECOME_METHODS).difference(('su',))

    def __init__(self, play_context, new_stdin, *args, **kwargs):
        super(Connection, self).__init__(play_context, new_stdin, *args, **kwargs)

        self.chroot = self._play_context.remote_addr

        #if os.geteuid() != 0:
            #raise AnsibleError("chroot connection requires running as root")

        # we're running as root on the local system so do some
        # trivial checks for ensuring 'host' is actually a chroot'able dir
        #if not os.path.isdir(self.chroot):
        #    raise AnsibleError("%s is not a directory" % self.chroot)

        #chrootsh = os.path.join(self.chroot, 'bin/sh')
        ## Want to check for a usable bourne shell inside the chroot.
        ## is_executable() == True is sufficient.  For symlinks it
        ## gets really complicated really fast.  So we punt on finding that
        ## out.  As long as it's a symlink we assume that it will work
        #if not (is_executable(chrootsh) or (os.path.lexists(chrootsh) and os.path.islink(chrootsh))):
        #    raise AnsibleError("%s does not look like a chrootable dir (/bin/sh missing)" % self.chroot)

        self.chroot_cmd = distutils.spawn.find_executable('schroot')
        if not self.chroot_cmd:
            raise AnsibleError("chroot command not found in PATH")

        self.chroot_id = "session:" + self.chroot
        self.chroot_type = "stable"
        existing = subprocess.check_output([self.chroot_cmd, "--list", "--all"])
        self.chroot_exists = False
        for line in existing.splitlines():
            if line == self.chroot_id:
                self.chroot_exists = True

    def set_host_overrides(self, host, hostvars=None):
        super(Connection, self).set_host_overrides(host, hostvars)
        self.chroot_type = hostvars.get("schroot_type", self.chroot_type)

    def _connect(self):
        ''' connect to the chroot; nothing to do here '''
        super(Connection, self)._connect()
        if not self._connected:
            if not self.chroot_exists:
                self.chroot_id = subprocess.check_output([self.chroot_cmd, "-b", "-c", self.chroot_type, "-n", self.chroot]).strip()
                subprocess.check_call([self.chroot_cmd, "-r", "-c", self.chroot_id, "-u", "root", "--", "apt-get", "update"])
                subprocess.check_call([self.chroot_cmd, "-r", "-c", self.chroot_id, "-u", "root", "--", "apt-get", "-y", "install", "python"])

            display.vvv("THIS IS A LOCAL CHROOT DIR", host=self.chroot)
            self._connected = True

    def _buffered_exec_command(self, cmd, stdin=subprocess.PIPE):
        ''' run a command on the chroot.  This is only needed for implementing
        put_file() get_file() so that we don't have to read the whole file
        into memory.

        compared to exec_command() it looses some niceties like being able to
        return the process's exit code immediately.
        '''
        executable = C.DEFAULT_EXECUTABLE.split()[0] if C.DEFAULT_EXECUTABLE else '/bin/sh'
        local_cmd = [self.chroot_cmd, "-r", "-c", self.chroot_id, "-u", "root", "--", executable, '-c', cmd]

        display.vvv("EXEC %s" % (local_cmd), host=self.chroot)
        local_cmd = [to_bytes(i, errors='strict') for i in local_cmd]
        p = subprocess.Popen(local_cmd, shell=False, stdin=stdin,
                stdout=subprocess.PIPE, stderr=subprocess.PIPE)

        return p

    def exec_command(self, cmd, in_data=None, sudoable=False):
        ''' run a command on the chroot '''
        super(Connection, self).exec_command(cmd, in_data=in_data, sudoable=sudoable)

        p = self._buffered_exec_command(cmd)

        stdout, stderr = p.communicate(in_data)
        return (p.returncode, stdout, stderr)

    def _prefix_login_path(self, remote_path):
        ''' Make sure that we put files into a standard path

            If a path is relative, then we need to choose where to put it.
            ssh chooses $HOME but we aren't guaranteed that a home dir will
            exist in any given chroot.  So for now we're choosing "/" instead.
            This also happens to be the former default.

            Can revisit using $HOME instead if it's a problem
        '''
        if not remote_path.startswith(os.path.sep):
            remote_path = os.path.join(os.path.sep, remote_path)
        return os.path.normpath(remote_path)

    def put_file(self, in_path, out_path):
        ''' transfer a file from local to chroot '''
        super(Connection, self).put_file(in_path, out_path)
        display.vvv("PUT %s TO %s" % (in_path, out_path), host=self.chroot)

        out_path = pipes.quote(self._prefix_login_path(out_path))
        try:
            with open(to_bytes(in_path, errors='strict'), 'rb') as in_file:
                try:
                    p = self._buffered_exec_command('dd of=%s bs=%s' % (out_path, BUFSIZE), stdin=in_file)
                except OSError:
                    raise AnsibleError("chroot connection requires dd command in the chroot")
                try:
                    stdout, stderr = p.communicate()
                except:
                    traceback.print_exc()
                    raise AnsibleError("failed to transfer file %s to %s" % (in_path, out_path))
                if p.returncode != 0:
                    raise AnsibleError("failed to transfer file %s to %s:\n%s\n%s" % (in_path, out_path, stdout, stderr))
        except IOError:
            raise AnsibleError("file or module does not exist at: %s" % in_path)

    def fetch_file(self, in_path, out_path):
        ''' fetch a file from chroot to local '''
        super(Connection, self).fetch_file(in_path, out_path)
        display.vvv("FETCH %s TO %s" % (in_path, out_path), host=self.chroot)

        in_path = pipes.quote(self._prefix_login_path(in_path))
        try:
            p = self._buffered_exec_command('dd if=%s bs=%s' % (in_path, BUFSIZE))
        except OSError:
            raise AnsibleError("chroot connection requires dd command in the chroot")

        with open(to_bytes(out_path, errors='strict'), 'wb+') as out_file:
            try:
                chunk = p.stdout.read(BUFSIZE)
                while chunk:
                    out_file.write(chunk)
                    chunk = p.stdout.read(BUFSIZE)
            except:
                traceback.print_exc()
                raise AnsibleError("failed to transfer file %s to %s" % (in_path, out_path))
            stdout, stderr = p.communicate()
            if p.returncode != 0:
                raise AnsibleError("failed to transfer file %s to %s:\n%s\n%s" % (in_path, out_path, stdout, stderr))

    def close(self):
        ''' terminate the connection; nothing to do here '''
        super(Connection, self).close()
        #subprocess.check_command([self.chroot_cmd, "-e", "-c", self.chroot_id])
        self._connected = False
debian eng pdo sw
2016-07-09 19:23:35+02:00

Monthly link collections with staticsite

A year ago, I wrote:

Instead of keeping substantial tabs open until I have read all of them, or losing them in the jungle of browser bookmarks, I have written a script that collects them into a file per month, and turns them into markdown files for my blog.

That script turned out to be quirky and overengineered, so much so that I stopped using it myself.

I've now rethought my approach, and downscaled it: instead of saving a copy of each page locally, I can blog a reference to https://archive.org or https://archive.is. I do not need to autogenerate a description from the site itself.

The result has been a nicely minimal set of changes to staticsite that resulted in a new version where adding a link to a monthly collection is as easy as typing ssite new -a links.

As long as I'll remember to rebuild the site 3 weeks from now, a new post should automagically appear in my blog.

eng pdo ssite sw
2016-06-15 21:47:28+02:00

Verifying gpg keys

Suppose you have a gpg keyid like 9F6C6333 that corresponds to both key 1AE0322EB8F74717BDEABF1D44BB1BA79F6C6333 and 88BB08F633073D7129383EE71EA37A0C9F6C6333, and you don't know which of the two to use.

You go to http://pgp.cs.uu.nl/ and find out that the site uses short key IDs, so the two keys are indistinguishable.

Building on Clint's hopenpgp-tools, I made a script that screenscrapes http://pgp.cs.uu.nl/ for trust paths, downloads all the potentially connecting keys in a temporary keyring, and runs hkt findpaths on it:

$ ./verify-trust-paths 1793D6AB75663E6BF104953A634F4BD1E7AD5568 1AE0322EB8F74717BDEABF1D44BB1BA79F6C6333
hkt (hopenpgp-tools) 0.18
Copyright (C) 2012-2016  Clint Adams
hkt comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions.
(4,[1,4,3,6])

(1,1793D6AB75663E6BF104953A634F4BD1E7AD5568)
(3,F8921D3A7404C86E11352215C7197699B29B232A)
(4,C331BA3F75FB723B5873785B06EAA066E397832F)
(6,1AE0322EB8F74717BDEABF1D44BB1BA79F6C6333)

$ ./verify-trust-paths 1793D6AB75663E6BF104953A634F4BD1E7AD5568 88BB08F633073D7129383EE71EA37A0C9F6C6333
hkt (hopenpgp-tools) 0.18
Copyright (C) 2012-2016  Clint Adams
hkt comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions.
(0,[])

This is a start: it could look in the local keyring for all ultimately trusted key finegrprints and use those as starting points. It could just take as an argument a short keyid and automatically check all matching fingerprints.

I'm currently quite busy with https://nm.debian.org and at the moment verify-trust-paths scratches enough of my itch that I can move on with my other things.

Please send patches, or take it over: I'd like to see this grow.

debian eng pdo sw
2016-03-17 16:28:03+01:00

Enhanced xkcd password generator

gpw and xkcdpass have entered testing.

How about putting them together?

#!/usr/bin/python3

import subprocess
import random
import re

re_spaces = re.compile(r"\s+")

def get_words(cmd):
    out = subprocess.check_output(cmd, universal_newlines=True)
    return (x for x in re_spaces.split(out) if x)

words = set()

words.update(get_words(["xkcdpass", "-c", "10"]))

for l in range(6, 9):
    words.update(get_words(["gpw", "10", str(l)]))

for i in range(7):
    print(" ".join(random.sample(words, 5)))

Let's check the results:

#!/usr/bin/python

from __future__ import print_function
from zxcvbn import password_strength
import sys

for line in sys.stdin:
    pw = line.strip()
    info = password_strength(pw)
    print("{pw}: {crack_time_display} {entropy}".format(pw=pw, **info))
$ xkcdpass -n 5 -c 7 | ./checkpass
moribund fasten cologne enamor by-and-by: centuries 132.586
woolen leptonic pogrom roster Serra: centuries 127.145
shorthand Piman suctional elution unholy: centuries 119.809
Blackwell mauvish covey selenium finished: centuries 112.936
asthma bloat crenulate clean-cut cytoplasm: centuries 150.181
agreeable refined racism Captain Cyrenaic: centuries 104.28
Bothnia stagnant atomist these revise: centuries 97.91
$ gpw 7 | ./checkpass
ncenspri: 11.0 days 33.903
orsterma: instant 19.54
entsssel: 14.0 hours 29.788
tempispi: instant 20.058
nidgersi: 14.0 hours 29.725
oligalin: 51.0 minutes 25.824
iseartio: 10.0 minutes 23.207
$ ./metapass | ./checkpass
herups icirop overdub oldster cingsape: centuries 147.265
mujahidin usufruct rition porky prignam: centuries 153.684
cannula irenics tableaux ntshed calcific: centuries 152.196
stopgap stale orthente Amazonia sharer: centuries 113.474
edudives mmingsso eremab prignam tableaux: centuries 147.354
tingei heroism Nathaniel cannula molasses: centuries 103.007
Torres blood lewdness prignam eremab: centuries 118.475

The strength, according to zxcvbn, seems only marginally better, and it's a wonderful way to grow a vocabulary of nonexisting words that can be dropped into cheap talk conversation to confuse the innocents.

debian eng pdo sw
2016-03-16 10:16:12+01:00

Postprocessing files saved by vim

Processing vim buffers through an external program on save, is a technique that is probably in use, and that I have not found documented anywhere, so here we go. Skip to the details for the entertaining example.

I wanted to integrate egt with TaskWarrior so that every time I am editing an egt project file in vim, if I write a line in a certain way then a new taskwarrior task is created. Also, I have never written a vim plugin, and I don't want to dive into another rabbit hole just yet.

Lynoure brought to my attention that taskwiki processes things when the file is saved, so I thought that I can probably get a long way by piping the file from vim into egt and reading it back before saving.

I created a new egt annotate command that reads a project file, tweaks it, and then prints it out; I opened a project file in vim; I typed :%!egt annotate %:p --stdin and saw that it could be done.

I find the result quite fun: I type 15 March: 18:00-21:00 in a log in egt, save the file, and it becomes 15 March: 18:00-21:00 3h. I do something in TaskWarrior, save the file in egt, and the lines that are TaskWarrior tasks update with the new task states.

Details

Here's a step by step example of how to hook into vim in this way.

First thing, create a filter script that processes text files in some way. Let's call this /usr/local/bin/pyvimshell:

#!/usr/bin/python3

import sys
import subprocess

for line in sys.stdin:
    print(line.rstrip(), file=sys.stdout)

line = line.strip()
if line == "$":
    pass
elif line.startswith("$"):
    out = subprocess.check_output(["sh", "-c", line[1:].strip()], universal_newlines=True)
    sys.stdout.write(out)
    if not out.endswith("\n"):
        sys.stdout.write("\n")
    print("$ ", file=sys.stdout)

Then let's create a new filetype in ~/.vim/filetype.vim for our special magic text files:

if exists("did_load_filetypes")
  finish
endif

augroup filetypedetect
  au! BufNewFile,BufRead *.term setf SillyShell
augroup END

If you create a file example.term, open it in vim and type :set ft it should say SillyShell.

Finally, the hook in ~/.vim/after/ftplugin/SillyShell.vim:

function! SillyShellRun()
    :%!/home/enrico/dev/egt/pyvimshell
    :$
endfunction
autocmd BufWritePre,FileWritePre <buffer> :silent call SillyShellRun()

Now you can create a file example.term, open it in vim, type $ ls, save it, and suddenly you have a terminal with infinite scrollback.

For egt, I actually want to preserve my cursor position across saves, and egt also needs to know the path to the project file, so here is the egt version of the hook:

function! EgtAnnotate()
    let l:cur_pos = getpos(".")
    :%!egt annotate --stdin %:p
    call setpos(".", l:cur_pos)
endfunction
autocmd BufWritePre,FileWritePre <buffer> :silent call EgtAnnotate()

I find this adorably dangerous. Have fun.

debian eng pdo sw
2016-03-06 19:16:06+01:00

Live preview of Markdown documentation

At work, to simplify build dependencies of DB-All.e we decided to port the documentation from LaTeX to Markdown.

Shortly after starting with the porting I resented not having a live preview of my work. I guess I got addicted to it with staticsite.

Actually, staticsite does preview interlinked Markdown files. I wonder if GitHub supports cross-linking between Markdown files in the same repo? It does, and incidentally it uses the same syntax as staticfile.

It shouldn't take long to build a different front-end on top of the staticsite engine just for this purpose. Indeed it didn't take long: here it is: mdpreview.

So, as you are editing the README.md of your project, you can now run mdpreview in the project directory, and you get live preview on your browser. If your README.md links to other documentation in your project, those links will work, too.

mdpreview uses the same themes as staticsite, so you can even tweak its appearance. And if you need to render the documentation and put it online somewhere, then staticsite can render it for you.

I experimented with it splitting staticsite's documentation into several parts, a I had great fun with it.

So, you want live preview of your project's Markdown documentation? mdpreview

When you are happy with it you can commit it to GitHub and it will show just fine.

You want it to show on your website instead? Build it with staticsite.

I'm considering merging staticsite and mdpreview somehow. Maybe mdpreview could just be a different command line front-end to staticsite's functionality. That's food for though for the next days.

Would you prefer to preview something else instead of Markdown? There is actually nothing markup specific in staticsite, so you can take this file as inspiration and implement support for the markup language of your choice in this whole toolchain. Except maybe for GitHub's website: that doesn't run on staticsite (yet).

eng pdo ssite sw
2016-03-04 11:54:45+01:00

Praise of component reuse

I farm bits and pieces out to the guys who are much more brilliant than I am. I say, "build me a laser", this. "Design me a molecular analyzer", that. They do, and I just stick 'em together. (Seth Brundle, "The Fly")

When I decided to try and turn siterefactor into staticsite, I decided that I would go ahead only for as long as it could be done with minimal work, writing code in the most straightforward way on top of existing and stable components.

I am pleased by how far that went.

Python-Markdown

It works fast enough, already comes with extensions for most of what I needed, and can be extended in several ways.

One of the extension methods is a hook for manipulating the ElementTree of the rendered document before serializing it to HTML, which made it really easy to go and process internal links in all <a href= and <img src= attributes.

To tell an internal link from an external link I just use the standard python urlparse and see if the link has a scheme or a netloc component. If it does not, and if it has a path, then it is an internal link.

This also means that I do not need to invent new Markdown syntax for internal references, avoiding the need for remembering things like [text]({{< relref "blog/post.md" >}}) or [text]({filename}/blog/post.md). In staticsite, it's just [text](/blog/post.md) or [text](post.md) if the post is nearby.

This feels nicely clean to me: if I wanted to implement fancy markdown features, I could do it as Python-Markdown extensions and submit them upstream. If I wanted to implement fancy interlinking features, I could do it with a special url scheme in links.

For example, it would be straigtforward to implement a ssite: url scheme that expanded the url with elements from staticsite's settings using a call to python's string.format (ssite:{SETTING_NAME}/bar maybe?), except I do not currently see any use cases for extending internal linking from what it is now.

Jinja2

Jina2 is a template engine that I already knew, it is widely used, powerful and pleasant to use, both on the templating side and on the API's side.

It is not HTML specific, so I can also use it to generate Atom, RSS2, "dynamic" site content, and even new site Markdown pages.

Implementing RSS and Atom feeds was just a matter of writing and testing these Jinja2 macros and then reusing them anywhere.

toml, yaml, json

No need to implement my own front matter parsing. Also, reusing the same syntax as Hugo allows me to just link to its documentation.

python-slugify

I found python-slugify so I did not bother writing a slug-generating function.

As a side effect, now things works better than I would even have thought to implement, including transliteration of non-ascii characters:

$ ./ssite new example --noedit --title "Cosí parlò Enrico"
/enrico-dev/staticsite/example/site/blog/2016/cosi-parlo-enrico.md

(I just filed an RFP)

python-livereload

Implementing ssite serve which monitors the file system and autoreloads when content changes and renders everything on the fly, took about an hour. Most of that hour went into implementing rendering pages on demand.

Then I discovered that it autoreloads even when I edit staticsite's source code.

Then I discovered that it communicates with the browser and even automatically triggers a page refresh.

I can keep vim on half my screen and a browser in the other half, and I get live preview for free every time I save, without ever leaving the editor.

Bootstrap

I already use Bootstrap at work, so creating the default theme templates with it took about 10 minutes.

This morning I tried looking at my website using my mobile phone, and I pleasantly saw it automatically turning into a working mobile version of itself.

Pygments

Python-Markdown uses Pygments for syntax highlighting, and it can be themed just by loading a .css.

So, without me really doing anything, even staticsite's syntax highligthing is themable, and there's even a nice page with a list of themes to choose from.

Everything else...

Command line parsing? Straight argparse.

Logging? python's logging support.

Copying static resource files? shutil.copy2.

Parsing dates? dateutil.parser.

Timing execution? time.perf_counter.

Timezone handling? pytz.

Building the command to run an editor? string.format.

Matching site pages? fnmatch.translate.

...and then some.

If I ever decide to implement incremental rendering, how do I implement tracking which source files have changed?

Well, for example, how about just asking git?

eng pdo ssite sw
2016-03-01 13:36:15+01:00

Static site generators

I decided to rethink the state of my personal site, and try out some of the new static site generators that are available now.

To do that, I jotted down a series of things that I want in a static site generator, then wrote a tool to convert my ikiwiki site to other formats, and set out to evaluate things.

As a benchmark I did a full rebuild of my site, which currently contains 1164 static files and 458 markdown pages.

My requirements

Free layout for my site

My / is mostly a high-level index to the site contents.

Blog posts are at /blog.

My talk archive is organised like a separate blog at /talks.

I want the freedom to create other sections of the site, each with its own rss feed, located wherever I want in the site hierarchy.

Assets next to posts

I occasionally blog just a photo with a little comment, and I would like the .md page with the comment to live next to the image in the file system.

I did not even know that I had this as a requirement until I found static site generators that mandated a completely different directory structure for markdown contents and for static assets.

Multiple RSS/Atom feeds

I want at least one RSS/Atom feed per tag, because I use tags for marking which articles go to http://planet.debian.org.

I also want RSS/Atom feeds for specific parts of the site, like the blog and talks.

Arbitrary contents in /index.html

I want total control over the contents of the main home page of the site.

Quick preview while editing contents

I do not like to wait several seconds for the site to be rebuilt at every review iteration of the pages I write.

This makes me feel like the task of editing is harder than it should, and makes me lose motivation to post.

Reasonable time for a full site rebuild

I want to be able to run a full rebuild of the site in a reasonable time.

I could define "reasonable" in this case as how long I can stare at the screen without getting bored, starting to do something else, and forgetting what it was that I was doing with the site.

It is ok if a rebuild takes something like 10 or 30 seconds. It is not ok if it takes minutes.

Code and dependency ecosystems that I can work with

I can deal with Python and Go.

I cannot deal with Ruby or JavaScript.

I forgot all about Perl.

Also, if it isn't in Debian it does not exist.

Decent themes out of the box

One of my hopes in switching to a more mainstream generator is to pick and choose themes and easily give my site a more modern look.

Hugo

Hugo is written in Go and is in Debian testing.

Full rebuild time for my site is acceptable, and it can even parallelize:

  $ time hugo
  real      0m5.285s
  user      0m9.556s
  sys       0m1.052s

Free layout for my site was hard to get.

I could replace /index.html by editing the template page for it, but then I did not find out how to create another similar index in an arbitrary place.

Also, archetypes are applied only on the first path component of new posts, but I would like them instead to be matched on the last path component first, and failing that traveling up to the path until the top. This should be easy to fix by reorganizing the content a bit around here

For example, a path for a new blog post of mine could be blog/2016/debian/ and I would like it to match the debian archetype first, and failing that the blog archetype.

Assets next to posts almost work.

Hugo automatically generates one feed per taxonomy element, and one feed per section. This would be currently sufficient for me, although I don't like the idea that sections map 1 to 1 to toplevel directories in the site structure.

Hugo has a server that watches the file system and rerenders pages as they are modified, so the quick preview while editing works fine.

About themes, it took me several tries to find a theme that would render navigation elements for both sections and tags, and most themes would render by pages with white components all around, and expect me to somehow dig in and tweak them. That frustrated me, because for quite a while I could not tell if I had misconfigured Hugo's taxonomies or if the theme was just somehow incomplete.

Nikola

Nikola is written in Python and is in Debian testing.

Full rebuild time for my site is almost two orders of magnitude more than Hugo, and I am miffed to find the phrases "Nikola is fast." or "Fast building process" in its front page and package description:

  $ time nikola build
  real      3m31.667s
  user      3m4.016s
  sys       0m24.684s

Free layout could be achieved fiddling with the site configuration to tell it where to read sources.

Assets next to post work after tweaking the configuration, but they require to write inconsistent links in the markdown source: https://github.com/getnikola/nikola/issues/2266 I have a hard time accepting that that, because I want to author content with consistent semantic interlinking, because I want to be able 10 years from now to parse it and convert it to something else if a new technology comes out.

Nikola generates one RSS/Atom feed per tag just fine. I have not tried generating feeds for different sections of the site.

Incremental generation inside its built in server works fine.

Pelican

Pelican is written in Python and is in Debian testing.

Full rebuild time for my site is acceptable:

  $ time pelican -d
  real      0m18.207s
  user      0m16.680s
  sys       0m1.448s

By default, pelican seems to put generate a single flat directory of html files regardless of the directory hierarchy of the sources. To have free layout, pelican needs some convincing in the configuration:

  PATH_METADATA = r"(?P<relpath>.+)\.md"
  ARTICLE_SAVE_AS = "{relpath}/index.html"

but even if I do that, the urls that it generates still point to just {slug}/index.html and I have not trivially found a configuration option to fix that accordingly. I got quite uncomfortable at the idea of needing to configure content generation and linking to match, instead of having one automatically being in sync with the other.

Having assets next to posts seems to be possible (also setting STATIC_PATHS = ["."]), but I do not recall making progress on this front.

I did not manage to generate a feed for each tag out of the box, and probably there is some knob in the configuration for it.

I gave up with Pelican as trying it out felt like a constant process of hacking the configuration from defaults that do not make any sense for me, withouth even knowing if a configuration exists that would do what I need

Ikiwiki

Ikiwiki is written in Perl and is in Debian. Although I am not anymore proficient with Perl, I was already using it, so it was worth considering.

Full rebuild time feels a bit on the slow side but is still acceptable:

  $ time ikiwiki --setup site.setup
  real      0m37.356s
  user      0m34.488s
  sys       0m1.536s

In terms of free site structure, all feeds for all or part of the site, ikiwiki just excels.

I even considered writing a python web server that monitors the file system and calls ikiwiki --refresh when anything changes, and calling it a day.

However, when I tried to re-theme my website around a simple bootstrap boilerplate, I found that to be hard, as a some of the HTML structure is hardcoded in Perl (and it's also my fault) and there is only so much that can be done by tweaking the (rather unreadable) templates.

siterefactor

During all these experiments I had built siterefactor to generate contents for all those static site engines, and it was going through all the contents quite fast:

  $ time ./siterefactor src dst -t hugo
  real  0m1.222s
  user  0m0.904s
  sys   0m0.308s

So I wondered how slow it would become if, instead of making it write markdown, I made it write HTML via python markdown and Jinja2:

  $ time ./siterefactor ~/zz/ikiwiki/pub/ ~/zz/ikiwiki/web -t web
  real  0m6.739s
  user  0m5.952s
  sys   0m0.416s

I then started wondering how slower it would become if I implemented postprocessing of all local URLs generated by Markdown to make sure they are kept consistent even if the path of a generated page is different than the path of its source. Not much slower, really.

I then added taxonomies. And arbitrary Jinja2 templates in the input, able to generate page lists and RSS/Atom feeds.

And theming.

And realised that reading all the sources and cross-linking them took 0.2 seconds, and the rest was generation time. And that after cross-linking, each page can be generated independently from all the others.

staticsite

So my site is now generated with staticsite:

  $ time ssite build
  real  0m6.833s
  user  0m5.804s
  sys   0m0.500s

It's comparable with Hugo, and on a single process.

eng pdo ssite sw