Enrico's blog

Last 10 blog posts

2016-05-23 14:45:34+02:00

I chipped in

I clicked on a random link and I found myself again in front of a wired.com popup that wanted to explain to me what I have to think about adblockers.

This time I was convinced, and I took my wallet out.

I finally donated $35 to AdBlock.

(And then somebody pointed me to uBlock Origin and I switched to that.)

eng pdo rant
2016-03-18 13:05:48+01:00

Adjusting with Moore's Law

In my last post I used zxcvbn to check the strength of passwords, and it gave estimates in hundreds of centuries.

However, there's Moore's law. If we assume that every 18 months the effective computing power will double...

#!/usr/bin/python3

import sys

years = int(sys.argv[1])

# How many 18 months intervals we spent
count = 0
# How many original-computation-time-months we compute each 18 months interval
step = 18
# How many original-computation-time-months we computed so far
total = 0
# How many original-computation-time-months we need
target = years * 12

# Iterate each 18 months interval
while True:
    count += 1
    total += step
    if total > target: break
    step *= 2

print("{:.2f} years needed".format(count / 12 * 18))

...then something that is now estimated in 152 centures is likely to take, well:

$ ./actual_years 15200
9.33 years needed

Update

Fixed algorithm (thanks John MacGregor). Now the estimate is a little longer, but still:

$ ./actual_years 15200
21.00 years needed

Łukasz Stelmach proposed an analytic solution:

There are two cars (remember phisics in primary school?). They both have the same speed V0. It is going to take "54.697 centuries" for car A to travel the desired distance at constant speed. Car B is accelerating, its speed is 2^(t/18m)*V0 (18 [m]onths).

s = V0 * T (car A, T is 54.697 centuries)

s = ∫2^(t/18)*V0 dt (car B, we're looking for t)

It's been some time but as far as I can tell it is

s = V0 * 2^(t/18) * 1/((1/18) * ln 2)

2^(t/18) * 1/((1/18) * ln 2) = 54.697 * 100 * 12 [months]

2^(t/18) = 2527.54

t = 18*log2(2527.54)

t = 16.96 years (still reasonable strength).

debian eng pdo
2016-03-17 16:28:03+01:00

Enhanced xkcd password generator

gpw and xkcdpass have entered testing.

How about putting them together?

#!/usr/bin/python3

import subprocess
import random
import re

re_spaces = re.compile(r"\s+")

def get_words(cmd):
    out = subprocess.check_output(cmd, universal_newlines=True)
    return (x for x in re_spaces.split(out) if x)

words = set()

words.update(get_words(["xkcdpass", "-c", "10"]))

for l in range(6, 9):
    words.update(get_words(["gpw", "10", str(l)]))

for i in range(7):
    print(" ".join(random.sample(words, 5)))

Let's check the results:

#!/usr/bin/python

from __future__ import print_function
from zxcvbn import password_strength
import sys

for line in sys.stdin:
    pw = line.strip()
    info = password_strength(pw)
    print("{pw}: {crack_time_display} {entropy}".format(pw=pw, **info))
$ xkcdpass -n 5 -c 7 | ./checkpass
moribund fasten cologne enamor by-and-by: centuries 132.586
woolen leptonic pogrom roster Serra: centuries 127.145
shorthand Piman suctional elution unholy: centuries 119.809
Blackwell mauvish covey selenium finished: centuries 112.936
asthma bloat crenulate clean-cut cytoplasm: centuries 150.181
agreeable refined racism Captain Cyrenaic: centuries 104.28
Bothnia stagnant atomist these revise: centuries 97.91
$ gpw 7 | ./checkpass
ncenspri: 11.0 days 33.903
orsterma: instant 19.54
entsssel: 14.0 hours 29.788
tempispi: instant 20.058
nidgersi: 14.0 hours 29.725
oligalin: 51.0 minutes 25.824
iseartio: 10.0 minutes 23.207
$ ./metapass | ./checkpass
herups icirop overdub oldster cingsape: centuries 147.265
mujahidin usufruct rition porky prignam: centuries 153.684
cannula irenics tableaux ntshed calcific: centuries 152.196
stopgap stale orthente Amazonia sharer: centuries 113.474
edudives mmingsso eremab prignam tableaux: centuries 147.354
tingei heroism Nathaniel cannula molasses: centuries 103.007
Torres blood lewdness prignam eremab: centuries 118.475

The strength, according to zxcvbn, seems only marginally better, and it's a wonderful way to grow a vocabulary of nonexisting words that can be dropped into cheap talk conversation to confuse the innocents.

debian eng pdo sw
2016-03-16 10:16:12+01:00

Postprocessing files saved by vim

Processing vim buffers through an external program on save, is a technique that is probably in use, and that I have not found documented anywhere, so here we go. Skip to the details for the entertaining example.

I wanted to integrate egt with TaskWarrior so that every time I am editing an egt project file in vim, if I write a line in a certain way then a new taskwarrior task is created. Also, I have never written a vim plugin, and I don't want to dive into another rabbit hole just yet.

Lynoure brought to my attention that taskwiki processes things when the file is saved, so I thought that I can probably get a long way by piping the file from vim into egt and reading it back before saving.

I created a new egt annotate command that reads a project file, tweaks it, and then prints it out; I opened a project file in vim; I typed :%!egt annotate %:p --stdin and saw that it could be done.

I find the result quite fun: I type 15 March: 18:00-21:00 in a log in egt, save the file, and it becomes 15 March: 18:00-21:00 3h. I do something in TaskWarrior, save the file in egt, and the lines that are TaskWarrior tasks update with the new task states.

Details

Here's a step by step example of how to hook into vim in this way.

First thing, create a filter script that processes text files in some way. Let's call this /usr/local/bin/pyvimshell:

#!/usr/bin/python3

import sys
import subprocess

for line in sys.stdin:
    print(line.rstrip(), file=sys.stdout)

line = line.strip()
if line == "$":
    pass
elif line.startswith("$"):
    out = subprocess.check_output(["sh", "-c", line[1:].strip()], universal_newlines=True)
    sys.stdout.write(out)
    if not out.endswith("\n"):
        sys.stdout.write("\n")
    print("$ ", file=sys.stdout)

Then let's create a new filetype in ~/.vim/filetype.vim for our special magic text files:

if exists("did_load_filetypes")
  finish
endif

augroup filetypedetect
  au! BufNewFile,BufRead *.term setf SillyShell
augroup END

If you create a file example.term, open it in vim and type :set ft it should say SillyShell.

Finally, the hook in ~/.vim/after/ftplugin/SillyShell.vim:

function! SillyShellRun()
    :%!/home/enrico/dev/egt/pyvimshell
    :$
endfunction
autocmd BufWritePre,FileWritePre <buffer> :silent call SillyShellRun()

Now you can create a file example.term, open it in vim, type $ ls, save it, and suddenly you have a terminal with infinite scrollback.

For egt, I actually want to preserve my cursor position across saves, and egt also needs to know the path to the project file, so here is the egt version of the hook:

function! EgtAnnotate()
    let l:cur_pos = getpos(".")
    :%!egt annotate --stdin %:p
    call setpos(".", l:cur_pos)
endfunction
autocmd BufWritePre,FileWritePre <buffer> :silent call EgtAnnotate()

I find this adorably dangerous. Have fun.

debian eng pdo sw
2016-03-08 20:16:40+01:00

Simple one liner to save battery life and reduce system latency

Sometimes I am in an airplane, and I just want to write code and compile it.

Sometimes I am playing a game in full screen, and I don't want it to lag.

Lynoure suggested this simple one-liner which helps immensely, to a point that I have turned it into a habit:

pkill -STOP chromium

When I want to use the browser again:

pkill -CONT chromium

I wish web browsers would stop running anything when unfocused, unless asked otherwise on a site by site basis.

Update: Johannes Schauer blogged more than a year ago about how to automate this in awesome.

debian eng pdo rant
2016-03-07 17:13:41+01:00

Polite invitations

It is DPL nomination time again.

A gentle reminder to avoid making people uncomfortable:

Inviting people to candidate for DPL is like inviting them for sex.

It can be flattering.

And if you insist after one says "no", it becomes harassment.

debian eng pdo
2016-03-06 19:16:06+01:00

Live preview of Markdown documentation

At work, to simplify build dependencies of DB-All.e we decided to port the documentation from LaTeX to Markdown.

Shortly after starting with the porting I resented not having a live preview of my work. I guess I got addicted to it with staticsite.

Actually, staticsite does preview interlinked Markdown files. I wonder if GitHub supports cross-linking between Markdown files in the same repo? It does, and incidentally it uses the same syntax as staticfile.

It shouldn't take long to build a different front-end on top of the staticsite engine just for this purpose. Indeed it didn't take long: here it is: mdpreview.

So, as you are editing the README.md of your project, you can now run mdpreview in the project directory, and you get live preview on your browser. If your README.md links to other documentation in your project, those links will work, too.

mdpreview uses the same themes as staticsite, so you can even tweak its appearance. And if you need to render the documentation and put it online somewhere, then staticsite can render it for you.

I experimented with it splitting staticsite's documentation into several parts, a I had great fun with it.

So, you want live preview of your project's Markdown documentation? mdpreview

When you are happy with it you can commit it to GitHub and it will show just fine.

You want it to show on your website instead? Build it with staticsite.

I'm considering merging staticsite and mdpreview somehow. Maybe mdpreview could just be a different command line front-end to staticsite's functionality. That's food for though for the next days.

Would you prefer to preview something else instead of Markdown? There is actually nothing markup specific in staticsite, so you can take this file as inspiration and implement support for the markup language of your choice in this whole toolchain. Except maybe for GitHub's website: that doesn't run on staticsite (yet).

eng pdo ssite sw
2016-03-04 11:54:45+01:00

Praise of component reuse

I farm bits and pieces out to the guys who are much more brilliant than I am. I say, "build me a laser", this. "Design me a molecular analyzer", that. They do, and I just stick 'em together. (Seth Brundle, "The Fly")

When I decided to try and turn siterefactor into staticsite, I decided that I would go ahead only for as long as it could be done with minimal work, writing code in the most straightforward way on top of existing and stable components.

I am pleased by how far that went.

Python-Markdown

It works fast enough, already comes with extensions for most of what I needed, and can be extended in several ways.

One of the extension methods is a hook for manipulating the ElementTree of the rendered document before serializing it to HTML, which made it really easy to go and process internal links in all <a href= and <img src= attributes.

To tell an internal link from an external link I just use the standard python urlparse and see if the link has a scheme or a netloc component. If it does not, and if it has a path, then it is an internal link.

This also means that I do not need to invent new Markdown syntax for internal references, avoiding the need for remembering things like [text]({{< relref "blog/post.md" >}}) or [text]({filename}/blog/post.md). In staticsite, it's just [text](/blog/post.md) or [text](post.md) if the post is nearby.

This feels nicely clean to me: if I wanted to implement fancy markdown features, I could do it as Python-Markdown extensions and submit them upstream. If I wanted to implement fancy interlinking features, I could do it with a special url scheme in links.

For example, it would be straigtforward to implement a ssite: url scheme that expanded the url with elements from staticsite's settings using a call to python's string.format (ssite:{SETTING_NAME}/bar maybe?), except I do not currently see any use cases for extending internal linking from what it is now.

Jinja2

Jina2 is a template engine that I already knew, it is widely used, powerful and pleasant to use, both on the templating side and on the API's side.

It is not HTML specific, so I can also use it to generate Atom, RSS2, "dynamic" site content, and even new site Markdown pages.

Implementing RSS and Atom feeds was just a matter of writing and testing these Jinja2 macros and then reusing them anywhere.

toml, yaml, json

No need to implement my own front matter parsing. Also, reusing the same syntax as Hugo allows me to just link to its documentation.

python-slugify

I found python-slugify so I did not bother writing a slug-generating function.

As a side effect, now things works better than I would even have thought to implement, including transliteration of non-ascii characters:

$ ./ssite new example --noedit --title "Cosí parlò Enrico"
/enrico-dev/staticsite/example/site/blog/2016/cosi-parlo-enrico.md

(I just filed an RFP)

python-livereload

Implementing ssite serve which monitors the file system and autoreloads when content changes and renders everything on the fly, took about an hour. Most of that hour went into implementing rendering pages on demand.

Then I discovered that it autoreloads even when I edit staticsite's source code.

Then I discovered that it communicates with the browser and even automatically triggers a page refresh.

I can keep vim on half my screen and a browser in the other half, and I get live preview for free every time I save, without ever leaving the editor.

Bootstrap

I already use Bootstrap at work, so creating the default theme templates with it took about 10 minutes.

This morning I tried looking at my website using my mobile phone, and I pleasantly saw it automatically turning into a working mobile version of itself.

Pygments

Python-Markdown uses Pygments for syntax highlighting, and it can be themed just by loading a .css.

So, without me really doing anything, even staticsite's syntax highligthing is themable, and there's even a nice page with a list of themes to choose from.

Everything else...

Command line parsing? Straight argparse.

Logging? python's logging support.

Copying static resource files? shutil.copy2.

Parsing dates? dateutil.parser.

Timing execution? time.perf_counter.

Timezone handling? pytz.

Building the command to run an editor? string.format.

Matching site pages? fnmatch.translate.

...and then some.

If I ever decide to implement incremental rendering, how do I implement tracking which source files have changed?

Well, for example, how about just asking git?

eng pdo ssite sw
2016-03-01 13:36:15+01:00

Static site generators

I decided to rethink the state of my personal site, and try out some of the new static site generators that are available now.

To do that, I jotted down a series of things that I want in a static site generator, then wrote a tool to convert my ikiwiki site to other formats, and set out to evaluate things.

As a benchmark I did a full rebuild of my site, which currently contains 1164 static files and 458 markdown pages.

My requirements

Free layout for my site

My / is mostly a high-level index to the site contents.

Blog posts are at /blog.

My talk archive is organised like a separate blog at /talks.

I want the freedom to create other sections of the site, each with its own rss feed, located wherever I want in the site hierarchy.

Assets next to posts

I occasionally blog just a photo with a little comment, and I would like the .md page with the comment to live next to the image in the file system.

I did not even know that I had this as a requirement until I found static site generators that mandated a completely different directory structure for markdown contents and for static assets.

Multiple RSS/Atom feeds

I want at least one RSS/Atom feed per tag, because I use tags for marking which articles go to http://planet.debian.org.

I also want RSS/Atom feeds for specific parts of the site, like the blog and talks.

Arbitrary contents in /index.html

I want total control over the contents of the main home page of the site.

Quick preview while editing contents

I do not like to wait several seconds for the site to be rebuilt at every review iteration of the pages I write.

This makes me feel like the task of editing is harder than it should, and makes me lose motivation to post.

Reasonable time for a full site rebuild

I want to be able to run a full rebuild of the site in a reasonable time.

I could define "reasonable" in this case as how long I can stare at the screen without getting bored, starting to do something else, and forgetting what it was that I was doing with the site.

It is ok if a rebuild takes something like 10 or 30 seconds. It is not ok if it takes minutes.

Code and dependency ecosystems that I can work with

I can deal with Python and Go.

I cannot deal with Ruby or JavaScript.

I forgot all about Perl.

Also, if it isn't in Debian it does not exist.

Decent themes out of the box

One of my hopes in switching to a more mainstream generator is to pick and choose themes and easily give my site a more modern look.

Hugo

Hugo is written in Go and is in Debian testing.

Full rebuild time for my site is acceptable, and it can even parallelize:

  $ time hugo
  real      0m5.285s
  user      0m9.556s
  sys       0m1.052s

Free layout for my site was hard to get.

I could replace /index.html by editing the template page for it, but then I did not find out how to create another similar index in an arbitrary place.

Also, archetypes are applied only on the first path component of new posts, but I would like them instead to be matched on the last path component first, and failing that traveling up to the path until the top. This should be easy to fix by reorganizing the content a bit around here

For example, a path for a new blog post of mine could be blog/2016/debian/ and I would like it to match the debian archetype first, and failing that the blog archetype.

Assets next to posts almost work.

Hugo automatically generates one feed per taxonomy element, and one feed per section. This would be currently sufficient for me, although I don't like the idea that sections map 1 to 1 to toplevel directories in the site structure.

Hugo has a server that watches the file system and rerenders pages as they are modified, so the quick preview while editing works fine.

About themes, it took me several tries to find a theme that would render navigation elements for both sections and tags, and most themes would render by pages with white components all around, and expect me to somehow dig in and tweak them. That frustrated me, because for quite a while I could not tell if I had misconfigured Hugo's taxonomies or if the theme was just somehow incomplete.

Nikola

Nikola is written in Python and is in Debian testing.

Full rebuild time for my site is almost two orders of magnitude more than Hugo, and I am miffed to find the phrases "Nikola is fast." or "Fast building process" in its front page and package description:

  $ time nikola build
  real      3m31.667s
  user      3m4.016s
  sys       0m24.684s

Free layout could be achieved fiddling with the site configuration to tell it where to read sources.

Assets next to post work after tweaking the configuration, but they require to write inconsistent links in the markdown source: https://github.com/getnikola/nikola/issues/2266 I have a hard time accepting that that, because I want to author content with consistent semantic interlinking, because I want to be able 10 years from now to parse it and convert it to something else if a new technology comes out.

Nikola generates one RSS/Atom feed per tag just fine. I have not tried generating feeds for different sections of the site.

Incremental generation inside its built in server works fine.

Pelican

Pelican is written in Python and is in Debian testing.

Full rebuild time for my site is acceptable:

  $ time pelican -d
  real      0m18.207s
  user      0m16.680s
  sys       0m1.448s

By default, pelican seems to put generate a single flat directory of html files regardless of the directory hierarchy of the sources. To have free layout, pelican needs some convincing in the configuration:

  PATH_METADATA = r"(?P<relpath>.+)\.md"
  ARTICLE_SAVE_AS = "{relpath}/index.html"

but even if I do that, the urls that it generates still point to just {slug}/index.html and I have not trivially found a configuration option to fix that accordingly. I got quite uncomfortable at the idea of needing to configure content generation and linking to match, instead of having one automatically being in sync with the other.

Having assets next to posts seems to be possible (also setting STATIC_PATHS = ["."]), but I do not recall making progress on this front.

I did not manage to generate a feed for each tag out of the box, and probably there is some knob in the configuration for it.

I gave up with Pelican as trying it out felt like a constant process of hacking the configuration from defaults that do not make any sense for me, withouth even knowing if a configuration exists that would do what I need

Ikiwiki

Ikiwiki is written in Perl and is in Debian. Although I am not anymore proficient with Perl, I was already using it, so it was worth considering.

Full rebuild time feels a bit on the slow side but is still acceptable:

  $ time ikiwiki --setup site.setup
  real      0m37.356s
  user      0m34.488s
  sys       0m1.536s

In terms of free site structure, all feeds for all or part of the site, ikiwiki just excels.

I even considered writing a python web server that monitors the file system and calls ikiwiki --refresh when anything changes, and calling it a day.

However, when I tried to re-theme my website around a simple bootstrap boilerplate, I found that to be hard, as a some of the HTML structure is hardcoded in Perl (and it's also my fault) and there is only so much that can be done by tweaking the (rather unreadable) templates.

siterefactor

During all these experiments I had built siterefactor to generate contents for all those static site engines, and it was going through all the contents quite fast:

  $ time ./siterefactor src dst -t hugo
  real  0m1.222s
  user  0m0.904s
  sys   0m0.308s

So I wondered how slow it would become if, instead of making it write markdown, I made it write HTML via python markdown and Jinja2:

  $ time ./siterefactor ~/zz/ikiwiki/pub/ ~/zz/ikiwiki/web -t web
  real  0m6.739s
  user  0m5.952s
  sys   0m0.416s

I then started wondering how slower it would become if I implemented postprocessing of all local URLs generated by Markdown to make sure they are kept consistent even if the path of a generated page is different than the path of its source. Not much slower, really.

I then added taxonomies. And arbitrary Jinja2 templates in the input, able to generate page lists and RSS/Atom feeds.

And theming.

And realised that reading all the sources and cross-linking them took 0.2 seconds, and the rest was generation time. And that after cross-linking, each page can be generated independently from all the others.

staticsite

So my site is now generated with staticsite:

  $ time ssite build
  real  0m6.833s
  user  0m5.804s
  sys   0m0.500s

It's comparable with Hugo, and on a single process.

eng pdo ssite sw
2016-03-01 11:54:00+01:00

I heard you like meta...

Now I want to write a blog post on blogging softwares, but first I need to write the bit of my blogging software that writes new blog posts.

How meta is the metadata on this post?

debian eng pdo sw