Tag sw

Latest posts for tag sw

2017-03-16 12:01:00+01:00

Django signing signs, does not encrypt

As is says in the documentation. django.core.signing signs, and does not encyrpt.

Even though signing.dumps creates obscure-looking tokens, they are not encrypted, and here's a proof:

>>> from django.core import signing
>>> a = signing.dumps({"action":"set-password", "username": "enrico", "password": "SECRET"})
>>> from django.utils.encoding import force_bytes
>>> print(signing.b64_decode(force_bytes(a.split(":",1)[0])))
b'{"action":"set-password","password":"SECRET","username":"enrico"}'

I'm writing it down so one day I won't be tempted to think otherwise.

debian django eng pdo sw
2017-02-22 14:10:58+01:00

staticsite news: github mode and post series

GitHub mode

Tobias Gruetzmacher implemented GitHub mode for staticsite.

Although GitHub now has a similar site rendering mode, it doesn't give you a live preview: if you run ssite serve on a GitHub project you will get a live preview of README.md and the project documentation.

Post series

I have added support for post series, that allow you to easily interlink posts with previous/next links.

You can see it in action on links and on An Italian song a day, an ongoing series that is currently each day posting a link to an Italian song.

eng pdo ssite sw
2016-12-25 13:38:33+01:00

"Intervallo RAI" generator

During holiday idling, I made a thing to generate picture slideshows similar to RAI's iconic "Intervallo"

You can get it at https://github.com/spanezz/intervallo

Usage:

$ intervallo --help
usage: intervallo [-h] [--font file.ttf] [--audio file.mp3] [--duration sec]
                  imgfile [imgfile ...]

Create an Intervallo RAI out of a collection of images.

positional arguments:
  imgfile           input image files

optional arguments:
  -h, --help        show this help message and exit
  --font file.ttf   Font to use for subtitles
  --audio file.mp3  Audio track
  --duration sec    Time for each image in seconds

For example:

./intervallo --font DejaVuSerif.ttf --audio Paradisi-Toccata.mp3 *.jpg

The images are captioned with their file name, without extension. You may want to rename the image files to have nice descriptive names.

For some audio to use, you can try https://archive.org/details/IntervalloRai-Paradisi

Example

https://www.youtube.com/watch?v=ZCERwjTB4ck

debian eng pdo sw
2016-11-09 10:10:42+01:00

On SPF

I woke up this morning with some Django server error mails in my inbox:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 9: ordinal not in range(128)
…
 'REMOTE_USER': u'…-guest@users.alioth.debian.org',

I did what one does in cases like these, I tried to fix the bug and mailed …-guest@users.alioth.debian.org asking them to try again and let me know if it works.

I get a bounce:

  <Actual user's email>
    (generated from …-guest@users.alioth.debian.org)
    SMTP error from remote mail server after MAIL FROM:<enrico@enricozini.org> SIZE=3948:
    host … […]: 550 Please see http://www.openspf.org/Why?id=enrico%40enricozini.org&ip=2001%3a41c8%3a1000%3a21%3a%3a21%3a21&receiver=bq :
    Reason: mechanism

I resent the mail to the actual user's address, and it went through. Job done, at least until they get back to me telling me that my fix didn't work.

Lessons learnt:

debian eng pdo sw
2016-11-02 16:54:06+01:00

schroot connector for ansible

I accidentally made an ansible connector plugin for schroot. I haven't even used ansible yet, so I have no idea what I am doing.

You can choose the chroot type by setting something like schroot_type: jessie in your variables.

You can install this locally as connection_plugins/schroot.py.

# Based on chroot.py (c) 2013, Maykel Moya <mmoya@speedyrails.com>
# Based on chroot.py (c) 2015, Toshio Kuratomi <tkuratomi@ansible.com>
# (c) 2016, Enrico Zini <enrico@debian.org>
#
# This is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Ansible is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

import distutils.spawn
import os
import os.path
import pipes
import subprocess
import traceback

from ansible import constants as C
from ansible.errors import AnsibleError
from ansible.plugins.connection import ConnectionBase, BUFSIZE
from ansible.module_utils.basic import is_executable
from ansible.utils.unicode import to_bytes

try:
    from __main__ import display
except ImportError:
    from ansible.utils.display import Display
    display = Display()


class Connection(ConnectionBase):
    ''' Local chroot based connections '''

    transport = 'schroot'
    has_pipelining = True
    # su currently has an undiagnosed issue with calculating the file
    # checksums (so copy, for instance, doesn't work right)
    # Have to look into that before re-enabling this
    become_methods = frozenset(C.BECOME_METHODS).difference(('su',))

    def __init__(self, play_context, new_stdin, *args, **kwargs):
        super(Connection, self).__init__(play_context, new_stdin, *args, **kwargs)

        self.chroot = self._play_context.remote_addr

        #if os.geteuid() != 0:
            #raise AnsibleError("chroot connection requires running as root")

        # we're running as root on the local system so do some
        # trivial checks for ensuring 'host' is actually a chroot'able dir
        #if not os.path.isdir(self.chroot):
        #    raise AnsibleError("%s is not a directory" % self.chroot)

        #chrootsh = os.path.join(self.chroot, 'bin/sh')
        ## Want to check for a usable bourne shell inside the chroot.
        ## is_executable() == True is sufficient.  For symlinks it
        ## gets really complicated really fast.  So we punt on finding that
        ## out.  As long as it's a symlink we assume that it will work
        #if not (is_executable(chrootsh) or (os.path.lexists(chrootsh) and os.path.islink(chrootsh))):
        #    raise AnsibleError("%s does not look like a chrootable dir (/bin/sh missing)" % self.chroot)

        self.chroot_cmd = distutils.spawn.find_executable('schroot')
        if not self.chroot_cmd:
            raise AnsibleError("chroot command not found in PATH")

        self.chroot_id = "session:" + self.chroot
        self.chroot_type = "stable"
        existing = subprocess.check_output([self.chroot_cmd, "--list", "--all"])
        self.chroot_exists = False
        for line in existing.splitlines():
            if line == self.chroot_id:
                self.chroot_exists = True

    def set_host_overrides(self, host, hostvars=None):
        super(Connection, self).set_host_overrides(host, hostvars)
        self.chroot_type = hostvars.get("schroot_type", self.chroot_type)

    def _connect(self):
        ''' connect to the chroot; nothing to do here '''
        super(Connection, self)._connect()
        if not self._connected:
            if not self.chroot_exists:
                self.chroot_id = subprocess.check_output([self.chroot_cmd, "-b", "-c", self.chroot_type, "-n", self.chroot]).strip()
                subprocess.check_call([self.chroot_cmd, "-r", "-c", self.chroot_id, "-u", "root", "--", "apt-get", "update"])
                subprocess.check_call([self.chroot_cmd, "-r", "-c", self.chroot_id, "-u", "root", "--", "apt-get", "-y", "install", "python"])

            display.vvv("THIS IS A LOCAL CHROOT DIR", host=self.chroot)
            self._connected = True

    def _buffered_exec_command(self, cmd, stdin=subprocess.PIPE):
        ''' run a command on the chroot.  This is only needed for implementing
        put_file() get_file() so that we don't have to read the whole file
        into memory.

        compared to exec_command() it looses some niceties like being able to
        return the process's exit code immediately.
        '''
        executable = C.DEFAULT_EXECUTABLE.split()[0] if C.DEFAULT_EXECUTABLE else '/bin/sh'
        local_cmd = [self.chroot_cmd, "-r", "-c", self.chroot_id, "-u", "root", "--", executable, '-c', cmd]

        display.vvv("EXEC %s" % (local_cmd), host=self.chroot)
        local_cmd = [to_bytes(i, errors='strict') for i in local_cmd]
        p = subprocess.Popen(local_cmd, shell=False, stdin=stdin,
                stdout=subprocess.PIPE, stderr=subprocess.PIPE)

        return p

    def exec_command(self, cmd, in_data=None, sudoable=False):
        ''' run a command on the chroot '''
        super(Connection, self).exec_command(cmd, in_data=in_data, sudoable=sudoable)

        p = self._buffered_exec_command(cmd)

        stdout, stderr = p.communicate(in_data)
        return (p.returncode, stdout, stderr)

    def _prefix_login_path(self, remote_path):
        ''' Make sure that we put files into a standard path

            If a path is relative, then we need to choose where to put it.
            ssh chooses $HOME but we aren't guaranteed that a home dir will
            exist in any given chroot.  So for now we're choosing "/" instead.
            This also happens to be the former default.

            Can revisit using $HOME instead if it's a problem
        '''
        if not remote_path.startswith(os.path.sep):
            remote_path = os.path.join(os.path.sep, remote_path)
        return os.path.normpath(remote_path)

    def put_file(self, in_path, out_path):
        ''' transfer a file from local to chroot '''
        super(Connection, self).put_file(in_path, out_path)
        display.vvv("PUT %s TO %s" % (in_path, out_path), host=self.chroot)

        out_path = pipes.quote(self._prefix_login_path(out_path))
        try:
            with open(to_bytes(in_path, errors='strict'), 'rb') as in_file:
                try:
                    p = self._buffered_exec_command('dd of=%s bs=%s' % (out_path, BUFSIZE), stdin=in_file)
                except OSError:
                    raise AnsibleError("chroot connection requires dd command in the chroot")
                try:
                    stdout, stderr = p.communicate()
                except:
                    traceback.print_exc()
                    raise AnsibleError("failed to transfer file %s to %s" % (in_path, out_path))
                if p.returncode != 0:
                    raise AnsibleError("failed to transfer file %s to %s:\n%s\n%s" % (in_path, out_path, stdout, stderr))
        except IOError:
            raise AnsibleError("file or module does not exist at: %s" % in_path)

    def fetch_file(self, in_path, out_path):
        ''' fetch a file from chroot to local '''
        super(Connection, self).fetch_file(in_path, out_path)
        display.vvv("FETCH %s TO %s" % (in_path, out_path), host=self.chroot)

        in_path = pipes.quote(self._prefix_login_path(in_path))
        try:
            p = self._buffered_exec_command('dd if=%s bs=%s' % (in_path, BUFSIZE))
        except OSError:
            raise AnsibleError("chroot connection requires dd command in the chroot")

        with open(to_bytes(out_path, errors='strict'), 'wb+') as out_file:
            try:
                chunk = p.stdout.read(BUFSIZE)
                while chunk:
                    out_file.write(chunk)
                    chunk = p.stdout.read(BUFSIZE)
            except:
                traceback.print_exc()
                raise AnsibleError("failed to transfer file %s to %s" % (in_path, out_path))
            stdout, stderr = p.communicate()
            if p.returncode != 0:
                raise AnsibleError("failed to transfer file %s to %s:\n%s\n%s" % (in_path, out_path, stdout, stderr))

    def close(self):
        ''' terminate the connection; nothing to do here '''
        super(Connection, self).close()
        #subprocess.check_command([self.chroot_cmd, "-e", "-c", self.chroot_id])
        self._connected = False
debian eng pdo sw
2016-07-09 19:23:35+02:00

Monthly link collections with staticsite

A year ago, I wrote:

Instead of keeping substantial tabs open until I have read all of them, or losing them in the jungle of browser bookmarks, I have written a script that collects them into a file per month, and turns them into markdown files for my blog.

That script turned out to be quirky and overengineered, so much so that I stopped using it myself.

I've now rethought my approach, and downscaled it: instead of saving a copy of each page locally, I can blog a reference to https://archive.org or https://archive.is. I do not need to autogenerate a description from the site itself.

The result has been a nicely minimal set of changes to staticsite that resulted in a new version where adding a link to a monthly collection is as easy as typing ssite new -a links.

As long as I'll remember to rebuild the site 3 weeks from now, a new post should automagically appear in my blog.

eng pdo ssite sw
2016-06-15 21:47:28+02:00

Verifying gpg keys

Suppose you have a gpg keyid like 9F6C6333 that corresponds to both key 1AE0322EB8F74717BDEABF1D44BB1BA79F6C6333 and 88BB08F633073D7129383EE71EA37A0C9F6C6333, and you don't know which of the two to use.

You go to http://pgp.cs.uu.nl/ and find out that the site uses short key IDs, so the two keys are indistinguishable.

Building on Clint's hopenpgp-tools, I made a script that screenscrapes http://pgp.cs.uu.nl/ for trust paths, downloads all the potentially connecting keys in a temporary keyring, and runs hkt findpaths on it:

$ ./verify-trust-paths 1793D6AB75663E6BF104953A634F4BD1E7AD5568 1AE0322EB8F74717BDEABF1D44BB1BA79F6C6333
hkt (hopenpgp-tools) 0.18
Copyright (C) 2012-2016  Clint Adams
hkt comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions.
(4,[1,4,3,6])

(1,1793D6AB75663E6BF104953A634F4BD1E7AD5568)
(3,F8921D3A7404C86E11352215C7197699B29B232A)
(4,C331BA3F75FB723B5873785B06EAA066E397832F)
(6,1AE0322EB8F74717BDEABF1D44BB1BA79F6C6333)

$ ./verify-trust-paths 1793D6AB75663E6BF104953A634F4BD1E7AD5568 88BB08F633073D7129383EE71EA37A0C9F6C6333
hkt (hopenpgp-tools) 0.18
Copyright (C) 2012-2016  Clint Adams
hkt comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions.
(0,[])

This is a start: it could look in the local keyring for all ultimately trusted key finegrprints and use those as starting points. It could just take as an argument a short keyid and automatically check all matching fingerprints.

I'm currently quite busy with https://nm.debian.org and at the moment verify-trust-paths scratches enough of my itch that I can move on with my other things.

Please send patches, or take it over: I'd like to see this grow.

debian eng pdo sw
2016-05-31 12:23:31+02:00

Python gotcha

This code:

#!/usr/bin/python3

class Test:
    def __init__(self, items=[]):
        self.items = items

    def add(self, item):
        self.items.append(item)


a = Test()
a.add("foo")
b = Test()
b.add("bar")
print(repr(a.items))
print(repr(b.items))

"obviously" prints:

['foo', 'bar']
['foo', 'bar']

Because the default value of the items argument is a mutable list constructed just once when the code is compiled when the function definition is evaluated, and then reused.

So, in Python, mutable items in default arguments are a good way to get more fun time with debugging.

devel eng pdo python sw
2016-03-17 16:28:03+01:00

Enhanced xkcd password generator

gpw and xkcdpass have entered testing.

How about putting them together?

#!/usr/bin/python3

import subprocess
import random
import re

re_spaces = re.compile(r"\s+")

def get_words(cmd):
    out = subprocess.check_output(cmd, universal_newlines=True)
    return (x for x in re_spaces.split(out) if x)

words = set()

words.update(get_words(["xkcdpass", "-c", "10"]))

for l in range(6, 9):
    words.update(get_words(["gpw", "10", str(l)]))

for i in range(7):
    print(" ".join(random.sample(words, 5)))

Let's check the results:

#!/usr/bin/python

from __future__ import print_function
from zxcvbn import password_strength
import sys

for line in sys.stdin:
    pw = line.strip()
    info = password_strength(pw)
    print("{pw}: {crack_time_display} {entropy}".format(pw=pw, **info))
$ xkcdpass -n 5 -c 7 | ./checkpass
moribund fasten cologne enamor by-and-by: centuries 132.586
woolen leptonic pogrom roster Serra: centuries 127.145
shorthand Piman suctional elution unholy: centuries 119.809
Blackwell mauvish covey selenium finished: centuries 112.936
asthma bloat crenulate clean-cut cytoplasm: centuries 150.181
agreeable refined racism Captain Cyrenaic: centuries 104.28
Bothnia stagnant atomist these revise: centuries 97.91
$ gpw 7 | ./checkpass
ncenspri: 11.0 days 33.903
orsterma: instant 19.54
entsssel: 14.0 hours 29.788
tempispi: instant 20.058
nidgersi: 14.0 hours 29.725
oligalin: 51.0 minutes 25.824
iseartio: 10.0 minutes 23.207
$ ./metapass | ./checkpass
herups icirop overdub oldster cingsape: centuries 147.265
mujahidin usufruct rition porky prignam: centuries 153.684
cannula irenics tableaux ntshed calcific: centuries 152.196
stopgap stale orthente Amazonia sharer: centuries 113.474
edudives mmingsso eremab prignam tableaux: centuries 147.354
tingei heroism Nathaniel cannula molasses: centuries 103.007
Torres blood lewdness prignam eremab: centuries 118.475

The strength, according to zxcvbn, seems only marginally better, and it's a wonderful way to grow a vocabulary of nonexisting words that can be dropped into cheap talk conversation to confuse the innocents.

debian eng pdo sw
2016-03-16 10:16:12+01:00

Postprocessing files saved by vim

Processing vim buffers through an external program on save, is a technique that is probably in use, and that I have not found documented anywhere, so here we go. Skip to the details for the entertaining example.

I wanted to integrate egt with TaskWarrior so that every time I am editing an egt project file in vim, if I write a line in a certain way then a new taskwarrior task is created. Also, I have never written a vim plugin, and I don't want to dive into another rabbit hole just yet.

Lynoure brought to my attention that taskwiki processes things when the file is saved, so I thought that I can probably get a long way by piping the file from vim into egt and reading it back before saving.

I created a new egt annotate command that reads a project file, tweaks it, and then prints it out; I opened a project file in vim; I typed :%!egt annotate %:p --stdin and saw that it could be done.

I find the result quite fun: I type 15 March: 18:00-21:00 in a log in egt, save the file, and it becomes 15 March: 18:00-21:00 3h. I do something in TaskWarrior, save the file in egt, and the lines that are TaskWarrior tasks update with the new task states.

Details

Here's a step by step example of how to hook into vim in this way.

First thing, create a filter script that processes text files in some way. Let's call this /usr/local/bin/pyvimshell:

#!/usr/bin/python3

import sys
import subprocess

for line in sys.stdin:
    print(line.rstrip(), file=sys.stdout)

line = line.strip()
if line == "$":
    pass
elif line.startswith("$"):
    out = subprocess.check_output(["sh", "-c", line[1:].strip()], universal_newlines=True)
    sys.stdout.write(out)
    if not out.endswith("\n"):
        sys.stdout.write("\n")
    print("$ ", file=sys.stdout)

Then let's create a new filetype in ~/.vim/filetype.vim for our special magic text files:

if exists("did_load_filetypes")
  finish
endif

augroup filetypedetect
  au! BufNewFile,BufRead *.term setf SillyShell
augroup END

If you create a file example.term, open it in vim and type :set ft it should say SillyShell.

Finally, the hook in ~/.vim/after/ftplugin/SillyShell.vim:

function! SillyShellRun()
    :%!/home/enrico/dev/egt/pyvimshell
    :$
endfunction
autocmd BufWritePre,FileWritePre <buffer> :silent call SillyShellRun()

Now you can create a file example.term, open it in vim, type $ ls, save it, and suddenly you have a terminal with infinite scrollback.

For egt, I actually want to preserve my cursor position across saves, and egt also needs to know the path to the project file, so here is the egt version of the hook:

function! EgtAnnotate()
    let l:cur_pos = getpos(".")
    :%!egt annotate --stdin %:p
    call setpos(".", l:cur_pos)
endfunction
autocmd BufWritePre,FileWritePre <buffer> :silent call EgtAnnotate()

I find this adorably dangerous. Have fun.

debian eng pdo sw