How to generate bootable USB keys with simple-cdd

simple-cdd is a lovely piece of software that builds a custom D-I image with the package selection and preseeding of your choice.

Today I was asked to build a bootable USB key with the simple-cdd image. Here is how; the general case is described in the d-i manual:

General USB key preparation:

1. download vmlinuz and initrd.img from hd-media
2. apt-get install syslinux mtools mbr
3. Partition the USB key as needed (from now on, I'll assume the usb key is in the device /dev/sdb1)
4. Format it as FAT: mkdosfs /dev/sdb1
5. Put the boot loader in it: syslinux /dev/sdb1
6. Put the MBR in it: install-mbr /dev/sdb
7. Mount it: mount /dev/sdb1 /mnt
8. Copy kernel and initrd: cp vmlinuz initrd.img /mnt/

simple-cdd specific part:

1. Run build-simple-cdd as usual
2. Copy the ISO file generated by build-simple-cdd in the USB key. Any name will do, as long as it ends in .iso the installer will find it
3. Configure the boot loader, fetching the kernel command line out of the cdrom boot loader generated by simple-cdd:
   • echo default vmlinuz > /mnt/syslinux.cfg
   • grep append tmp/cd-build/etch/boot1/isolinux/isolinux.cfg | head -1 | sed -e 's/^\t//' -e 's/ initrd=[^ ]*/ initrd=initrd.gz/' >> /mnt/syslinux.cfg

This is it, it works nicely, perfectly scriptable, tested today.

Setting environment variables at X login

I've been asked how to set a variable after gdm has done login. ~/.bashrc is not an option, as it's only run by shells, but we want the variable to be set in every X application that is started.

The answer is:

• ~/.profile for text and graphical sessions
• ~/.xprofile for graphical sessions only

Forget about ~/.xinitrc. ~/.xsession and ~/.Xsession: at least in gnome-session, they do not work.

Audit your debian uploads

My bank is sending me an e-mail every time I log into the home banking system, so that I can spot malicious logins.

My credit card is sending me a SMS message every time it gets charged, so that I can spot mailicious charges.

Can I get a notification of every Debian upload done with my key, so that I can spot if my key has been stolen?

Let's work on that. As a start, thanks to Ganneff, here is how to do a one-off audit:

# go to merkel to access projectb, which is the postgresql database
# with all dak information
$ ssh merkel
merkel$ psql projectb
# look up the database id of my fingerprint
projectb=> select id, fingerprint from fingerprint where fingerprint like '%797EBFAB';
 id  |               fingerprint                
    -----+------------------------------------------
     394 | 66B4DFB68CB24EBBD8650BC4F4B4B0CC797EBFAB
    (1 row)
# get a list of all uploads done with my key, sorted by date
projectb=> select * from source where sig_fpr=394 order by install_date desc;

First you get to do it (done); then you document it (done); then you automate it. It's quite trivial at this point, so enjoy the new Debian upload monitor.

It's got search as you type to find your full fingerprint, then you get an HTML page with the log of your uploads in the last 2 months, and the page has an RSS feed that you can use to track your own uploads.

Also, generating all this static content is acceptably fast:

merkel$ time ./deb-key-audit 

real    0m7.145s
user    0m4.244s
sys 0m0.384s

If you want to see the code, you can git clone http://merkel.debian.org/~enrico/keylog.git

Currently it wrongly encodes UTF-8 characters: I suppose the strings come out of the database as ASCII instead of UTF-8. A patch would be welcome to fix that.

I will now contact QA to see what we can do with it; if it ends up fitting in some bigger picture then it may be that the RSS links will change, but I'll post about it in that case.

How to not start a service by default

Use case: in my laptop, I sometimes need MySQL, PostgreSQL or Apache in order to test some software that I'm developing, but I do not want them on all the time.

The solution is: rm /etc/rc*.d/S*mysql* (thanks to Wouter)

update-rc.d will not touch your symlinks as long as there is at least one still around for a package. Also, this leaves the stop symlinks around, so that when I start one of these services for development, it will still be properly stopped on shutdown.

make distcheck and LaTeX

When building LaTeX documentation on a VPATH build, if your .tex file includes other files in the same directory, LaTeX will complain that it cannot find them. The reason is because in a VPATH build, latex is invoked like this:

latex ../../doc/manual.tex

What we need here is an equivalent to cc's -Idir for latex.

latex --help doesn't mention of such an option, nor of useful environment variables.

Googling a bit seems to suggest --include-directory=dir, but that gives me: unrecognized option '--include-directory=../../doc'

The manpage doesn't list commandline options. It however says:

The complete documentation for this version of TeX can be found in the info file or manual Web2C: A TeX implementation.

Without saying where that manual is, if it's installed and where, or what package installs it, or if instead should I look it up on the web.

info latex gives the manpage itself, of course.

Googling the title of that manual finds it, and it's a long one. Reading through, it points at the kpathsea manual, which then mentions you can set TEXINPUTS_latex, which however doesn't add but overrides, so your document will find the includes maybe, but not the LaTeX styles and other stuff.

But then later on it mentions that in the env variable you can use "default expansion", and it's another page of manual to read which tells you to put an extra colon in the end of the env var.

After half an hour of googling and trying things and cursing loud, here is the solution, which I hope will save others from this ugly search.

%.aux: %.tex
    TEXINPUTS="$(srcdir):" latex $<

# Oh, yes, and bibtex requires BIBINPUTS instead
%.bbl: %.aux
    BIBINPUTS="$(srcdir):" bibtex `basename $< .aux`

Fields used by an LDAP Unix user database

Some notes about LDAP for Unix user management:

Meaning of fields for objectClass posixAccount:

• uid contains the username
• uidNumber contains the numeric UID
• gidNumber should have the numeric GID
• cn should contain the user's full name (optional)
• homeDirectory and loginShell contain what you think they contain
• gecos contains the gecos from passwd (optional)
• userPassword contains {crypt} followed by the encrypted password from /etc/shadow (md5 password hashes are ok as well) ({SASL} is an interesting alternative)
• sn is the surname (optional)
• givenName is the given name (optional)

Meaning of fields for objectClass posixGroup:

• gidNumber is the group id
• cn is the group name
• memberUID attributes contain posixAccount.uid values

For objectClass inetOrgPerson: what you put there can be used as if it were a vCard by mail programs and contact lists.

Now, the name of people could potentially be split in cn,givenName,sn,gecos and displayName (possibly more): how would normal user tools deal with the redundancy? To show a gecos field, pam_ldap will search for a gecos field first, then automatically fallback on building a gecos field out of the other suitable info it finds. To show a name, sane programs try displayName first and if it's not present they guess using the rest.

Then there is the issue of how to chose the dn to identify users, groups and so on. Users usually go in uid=$USERNAME,ou=People,$SUFFIX, while groups would usually go in cn=$GROUPNAME,ou=Groups,$SUFFIX.

Should you need to create the People and Groups organizational units, this could be the proper bit of LDIF:

dn: ou=$NAME,$SUFFIX
ou: $NAME
objectClass: organizationalUnit

To add fields that are not already part of a schema, one needs to create their own schema. To do that, one needs to first obtain (free of charge) a Private Enterprise Number that is used in various places in the schema definition. Making up your own one means risking conflicts if you eventually grow larger. But it is rarely needed, because for most things there are already schemas available.

Many thanks to Wouter and noshadow for allowing me to crudely extract all this content from their brains.

Uploading gpsdrive tracks to openstreetmap

I've got some gpsdrive tracks and my area is blank on openstreetmap.

People pointed me at gpsbabel, and it took me a while to figure how it works. For the record, don't do any of this:

gpsbabel -i gpsdrive -o gpx -f track0000.sav -F track0000.gpx
gpsbabel -i gpsdrive -o gpx track0000.sav track0000.gpx

What you would have to do is this:

gpsbabel -i gpsdrive -f track0000.sav -o gpx -F track0000.gpx

However, it would choke on gpsdrive's "missing" points with all values set to 1001. You can grep them out, then gpsbabel would work, but openstreetmap would reject the data because the points have no timestamp: gpsbabel won't carry that on from the gpsdrive tracks to the GPX tracks.

The way to go is here, which contains a link to a tiny little perl script that will do the proper conversion for you:

./gpsdrive2gpx.pl track0000.sav > track0000.gpx

Those you can upload them to openstreetmap, at last.

Formatting numbers with iostream

What is the output of this C++ program?

#include <stdio.h>
#include <iostream>
#include <iomanip>

using namespace std;

main()
{
    printf("%05d\n", -3);
    cout << setw(5) << setfill('0') << -3 << endl;
}

It turns out that it is:

-0003
000-3

To get the numbers where you'd like them to be, you do it this way:

printf("%05d\n", -3);
cout << setw(5) << setfill('0') << internal -3 << endl;

Compare with C to see why C++ iostream is not really that much used. Also, setfill and internal are persistently changing the state of the stream, contributing fancy and interesting side effects:

cout << setw(5) << setfill('0') << internal << -3 << endl;
cout << setw(10) << "ciao" << endl;

this prints:

-0003
000000ciao

and I didn't see any sort of RAII save and restore of the stream state, nor a particularly obvious way to reset the stream to factory settings. Suppose that the function foo throws an exception here:

try {
    cout << setfill('=');
    cout << setw(10) << foo() << endl;
    cout << setfill(' ');
} catch (MyException& e) {
    // Catch this hypotetical exception
    cerr << setw(3) << e.errCode() << ": " << e.msg() << endl;
}

Then the state of the stream never gets reset, and the error code ends up being padded with equals instead of the default space. What would the alternative be, reset the stream state after each function call in the code? Or am I missing something?

Update: here are some suggested, but still not optimal, ways of dealing with it:

From Simon Richter:

std::cout << (std::stringstream() << "foo" << std::setw(8) << "bar").rdbuf() << std::endl;

From Peter De Wachter:

fstream tmp;
tmp.copyfmt(cout);
cout << setw(5) << setfill('0') << internal << -3 << endl;
cout.copyfmt(tmp);
cout << setw(10) << "ciao" << endl;

This is a possible RAII class to save the stream state (based on Peter De Watcher's approach), but I suspect that creating a fstream for every block that uses iomanip operators is rather heavyweight:

    struct SaveIOState
    {
            fstream tmp;
            ostream& s;
            SaveIOState(ostream& s) : s(s)
            {
                    tmp.copyfmt(s);
            }
            ~SaveIOState()
            {
                    s.copyfmt(tmp);
            }
    };

And this is a lightweight version, again thanks to Sylvain Sauvage:

class SaveIOState
{
    ios_base& s;
    ios_base::fmtflags f;
public:
    SaveIOState(ios_base& s) : s(s), f(s.flags())
    {}
    ~SaveIOState()
    {
        s.flags(f);
    }
};

Using a bluetooth headset for giving presentations

While looking for ways of flipping slides while not near the computer, I stumbled on this idea this idea, bought a cheap headset for like 10euros and started playing.

Getting the headset to work with alsa

It's supposed to just work, but bluez-utils in Debian has audio support disabled.

I've rebuild the package with the patch found in the bug report, copied the extra configuration files as I mentioned in a followup to the bug report, and brought my Debian up to speed.

Once the patched bluez-utils is in place, this is how to get the headset to work:

1. put the headset into pairing mode
2. hcitool scan (to get the address of the headset)

3. put this in ~/.asoundrc (using the address you got with hcitool scan):

   pcm.bluetooth_hw {
      type bluetooth
      device 00:11:22:33:44:55
   }
   
   
   ctl.bluetooth {
      type bluetooth
      device 00:11:22:33:44:55
   }
   
   
   # No idea why this is needed, but it makes aplay complain a bit
   # less, and also it makes aoss work
   pcm.bluetooth {
       type plug
       slave.pcm "bluetooth_hw"
   }
   
   

Now, forget hcitool cc: in order to pair with the headset, the command is:

mplayer -ao alsa:device=bluetooth moo.wav

This will trigger a funky chain of events that will eventually cause the bluetooth applet to ask you for the pin (0000 for the headsets), get things paired and activate super cow powers.

Now you have an alsa sound card called 'bluetooth' that works only with some, but not all, alsa applications. mplayer works. aoss works, but not always. aplay seems to work. twinkle won't work; apparently, it's because it tries to open the device multiple times.

Feeding synthesised speech to the headset

Synthesised speech is done by festival. Festival works with oss. The headset works with alsa. aoss works, but not always.

This is the script I'd like to run:

#!/bin/sh

warn() {
    echo -n "Attenzione: "
    if [ $1 == 1 ]
    then
        echo -n "manca un minuto"
    else
        echo -n "mancano $1 minuti"
    fi
    echo " alla fine."
}

(
    for x in 5 4 3 2 1
    do
        warn $x
        sleep 1
    done
) | festival --tts --language italian

By tweaking the sleep times and numbers a bit, it's a useful prototype that sends updates on how much time is left for the presentation.

This, under aoss, does not work reliably. An alternative is to replace the festival invocation with this:

echo string | text2wave tmpfile.wav
mplayer -ao alsa:device=bluetooth tmpfile.wav

However, I found no way to tell text2wave to use Italian instead of English.

I did manage, however, to get some link between festival and the headphones. At a further stage, I can write a program for the countdown that keeps the audio device open and every once in a while gets a waveform out of text2wave and feeds it to the audio device.

Keeping the audio device open seems a good idea, to avoid losing the beginning of the message while the connection is established (happens with my cheap headset) and to avoid the headset to go idle and switch off.

Reacting to the buttons on the headset

It's supposed to be possible, but I didn't manage to find out how to do this using the normal alsa driver, now that btsco is apparently deprecated.

Once I find out how to be notified of button presses, I'll try to see how to send a Page Down keystroke to OpenOffice Impress or xpdf.

Voice commands

sphinx2-bin seems promising, but since everyone around me is now asleep, playing with voice commands has to be postponed.

Further ideas

• IRC link: send me a message on a public IRC channel, and I could head it on the headphone: nice for giving talks that people is watching remotely via streaming.
• Giving the percentage of time passed, instead (or additionally to) the time remaining.
• Linking with the presentation software and also say the number of slides remaining.
• Get the status update being spoken not just at regular intervals, but also when changing slide.
• Get the headset to read slide notes, or only the slide notes marked with a special leading keyword.

mod_proxy_html and compressed pages

After putting it behind a reverse proxy, our phpmyadmin setup started showing empty pages.

After one morning of deep cursing, this is what happened:

1. the web server where phpmyadmin runs generates compressed html pages;
2. modproxyhtml tries to edit them, and "normalises" them, adding <html>...</html> headers around the compressed data;
3. Firefox fails to decompress because there is extra garbage, and shows a blank page instead of complaining.

Other things to note:

• firebug showed 4kb of data downloaded, but the page sources were empty;
• using curl to get the pages would show the data.

How I found it:

1. nc -l -p 444;
2. configure mod_proxy to send connections to netcat instead of the web browser;
3. compare curl headers and Firefox headers;
4. add the headers from Firefox to curl one by one, until the output breaks.

How to solve it:

1. a2enmod deflate
2. Replace SetOutputFilter proxy-html with SetOutputFilter DEFLATE;proxy-html;INFLATE so that we always have mod_proxy_html work on decompressed HTML.