How to generate bootable USB keys with simple-cdd

simple-cdd is a lovely piece of software that builds a custom D-I image with the package selection and preseeding of your choice.

Today I was asked to build a bootable USB key with the simple-cdd image. Here is how; the general case is described in the d-i manual:

General USB key preparation:

1. download vmlinuz and initrd.img from hd-media
2. apt-get install syslinux mtools mbr
3. Partition the USB key as needed (from now on, I'll assume the usb key is in the device /dev/sdb1)
4. Format it as FAT: mkdosfs /dev/sdb1
5. Put the boot loader in it: syslinux /dev/sdb1
6. Put the MBR in it: install-mbr /dev/sdb
7. Mount it: mount /dev/sdb1 /mnt
8. Copy kernel and initrd: cp vmlinuz initrd.img /mnt/

simple-cdd specific part:

1. Run build-simple-cdd as usual
2. Copy the ISO file generated by build-simple-cdd in the USB key. Any name will do, as long as it ends in .iso the installer will find it
3. Configure the boot loader, fetching the kernel command line out of the cdrom boot loader generated by simple-cdd:
   • echo default vmlinuz > /mnt/syslinux.cfg
   • grep append tmp/cd-build/etch/boot1/isolinux/isolinux.cfg | head -1 | sed -e 's/^\t//' -e 's/ initrd=[^ ]*/ initrd=initrd.gz/' >> /mnt/syslinux.cfg

This is it, it works nicely, perfectly scriptable, tested today.

Setting environment variables at X login

I've been asked how to set a variable after gdm has done login. ~/.bashrc is not an option, as it's only run by shells, but we want the variable to be set in every X application that is started.

The answer is:

• ~/.profile for text and graphical sessions
• ~/.xprofile for graphical sessions only

Forget about ~/.xinitrc. ~/.xsession and ~/.Xsession: at least in gnome-session, they do not work.

Update: * On IRC, I've been told that ~/.xsessionrc should be used since xorg 1:7.3+9

How to not start a service by default

Use case: in my laptop, I sometimes need MySQL, PostgreSQL or Apache in order to test some software that I'm developing, but I do not want them on all the time.

The solution is: rm /etc/rc*.d/S*mysql* (thanks to Wouter)

update-rc.d will not touch your symlinks as long as there is at least one still around for a package. Also, this leaves the stop symlinks around, so that when I start one of these services for development, it will still be properly stopped on shutdown.

OpenStreetMap party at Kaohsiung, Taiwan

Apparently, yesterday we had the first OpenStreetMap event in Taiwan!

We met in a café/restaurant equipped with power plug, wireless network and overhead projector and we had a bit of an introduction, chat and lunch.

Then we split in groups and exploited the fact that the newly built underground (KMRT) system is still free of charge, to spread around and map around the stations.

Finally, we reconvened at someone's house to see how to put the data together, draw roads, tag and upload.

Highlights of the day:

• How to turn a serial GPS into a data logger with 6 hours battery life . Then attach it to your bike using magnets from broken hard drives. Totally rocks!
• Previous OpenStreetMap data was collected by only one person, who took the fancy new High Speed Rail from the opposite side of the country and joined the party. This also made discussion about standardising tags for Taiwan rather easy.
• A group of people appeared wielding a number of "totally insane in every regard" Garmin GPSMAP units: it turns out they are with a civil action group that goes around mapping historical trails, abandoned railroads, aboriginal routes and mountain crosses and so on. Apparently, they did not know about OpenStreetMap: hopefully they'll join in.

Technical bits:

• The eeePC was very popular, and very handy for going around storing tracks, as you can just chuck it in one bag. JOSM runs fine, although it could really use an interface redesign to fit in the small screen. In fact, it could really use an interface redesign to fit in the standard 1024x768 screen of my laptop.
• We could not use the tracks made with the Garmins because we did not know we had to do "Setup -> Map -> Lock On Road = Off" and it was on by default. Now we know it for next time.
• Something like a SirfStarIII really helps in a city made mainly of very tall buildings with lots of steel and glass. My Sony-based cheap gps receiver that worked ok in the Bolognese countryside was next to useless here, continously losing the fix and producing a crazy zigzagging track of doom, only useful to figure out big long straight roads.
• Geocorrelation of digital camera pictures rocks! Who needs to store waypoints when you can just take pictures with the digital camera and have them show up as waypoints in JOSM? The trick of taking a picture of the GPS time and use that to compute time offset is great. Also, we found it easier to just fire up gpscorrelate to do the geocorrelation rather than figuring out how the tools in JOSM work.

Issues to address:

• There is a strong need for a zh_TW translation plugin of JOSM; I'll try to find out how to do it and pass on the information to who can do it.
• Road names could be written either in English or in Chinese characters. Currently English has been used for the name tag because osmarender cannot render Chinese characters. There is some planining to create an OSM mirror in Taiwan which renders twice, and allows to choose the rendering language for the map. I will try to get a planet.osm extract for Taiwan that people can use to experiment with this; thanks to people in #osm for giving me names of people to contact. I will try later after Europe wakes up from this even-earlier-than-usual sunday morning.

make distcheck and LaTeX

When building LaTeX documentation on a VPATH build, if your .tex file includes other files in the same directory, LaTeX will complain that it cannot find them. The reason is because in a VPATH build, latex is invoked like this:

latex ../../doc/manual.tex

What we need here is an equivalent to cc's -Idir for latex.

latex --help doesn't mention of such an option, nor of useful environment variables.

Googling a bit seems to suggest --include-directory=dir, but that gives me: unrecognized option '--include-directory=../../doc'

The manpage doesn't list commandline options. It however says:

The complete documentation for this version of TeX can be found in the info file or manual Web2C: A TeX implementation.

Without saying where that manual is, if it's installed and where, or what package installs it, or if instead should I look it up on the web.

info latex gives the manpage itself, of course.

Googling the title of that manual finds it, and it's a long one. Reading through, it points at the kpathsea manual, which then mentions you can set TEXINPUTS_latex, which however doesn't add but overrides, so your document will find the includes maybe, but not the LaTeX styles and other stuff.

But then later on it mentions that in the env variable you can use "default expansion", and it's another page of manual to read which tells you to put an extra colon in the end of the env var.

After half an hour of googling and trying things and cursing loud, here is the solution, which I hope will save others from this ugly search.

%.aux: %.tex
    TEXINPUTS="$(srcdir):" latex $<

# Oh, yes, and bibtex requires BIBINPUTS instead
%.bbl: %.aux
    BIBINPUTS="$(srcdir):" bibtex `basename $< .aux`

Fields used by an LDAP Unix user database

Some notes about LDAP for Unix user management:

Meaning of fields for objectClass posixAccount:

• uid contains the username
• uidNumber contains the numeric UID
• gidNumber should have the numeric GID
• cn should contain the user's full name (optional)
• homeDirectory and loginShell contain what you think they contain
• gecos contains the gecos from passwd (optional)
• userPassword contains {crypt} followed by the encrypted password from /etc/shadow (md5 password hashes are ok as well) ({SASL} is an interesting alternative)
• sn is the surname (optional)
• givenName is the given name (optional)

Meaning of fields for objectClass posixGroup:

• gidNumber is the group id
• cn is the group name
• memberUID attributes contain posixAccount.uid values

For objectClass inetOrgPerson: what you put there can be used as if it were a vCard by mail programs and contact lists.

Now, the name of people could potentially be split in cn,givenName,sn,gecos and displayName (possibly more): how would normal user tools deal with the redundancy? To show a gecos field, pam_ldap will search for a gecos field first, then automatically fallback on building a gecos field out of the other suitable info it finds. To show a name, sane programs try displayName first and if it's not present they guess using the rest.

Then there is the issue of how to chose the dn to identify users, groups and so on. Users usually go in uid=$USERNAME,ou=People,$SUFFIX, while groups would usually go in cn=$GROUPNAME,ou=Groups,$SUFFIX.

Should you need to create the People and Groups organizational units, this could be the proper bit of LDIF:

dn: ou=$NAME,$SUFFIX
ou: $NAME
objectClass: organizationalUnit

To add fields that are not already part of a schema, one needs to create their own schema. To do that, one needs to first obtain (free of charge) a Private Enterprise Number that is used in various places in the schema definition. Making up your own one means risking conflicts if you eventually grow larger. But it is rarely needed, because for most things there are already schemas available.

Many thanks to Wouter and noshadow for allowing me to crudely extract all this content from their brains.

Uploading gpsdrive tracks to openstreetmap

I've got some gpsdrive tracks and my area is blank on openstreetmap.

People pointed me at gpsbabel, and it took me a while to figure how it works. For the record, don't do any of this:

gpsbabel -i gpsdrive -o gpx -f track0000.sav -F track0000.gpx
gpsbabel -i gpsdrive -o gpx track0000.sav track0000.gpx

What you would have to do is this:

gpsbabel -i gpsdrive -f track0000.sav -o gpx -F track0000.gpx

However, it would choke on gpsdrive's "missing" points with all values set to 1001. You can grep them out, then gpsbabel would work, but openstreetmap would reject the data because the points have no timestamp: gpsbabel won't carry that on from the gpsdrive tracks to the GPX tracks.

The way to go is here, which contains a link to a tiny little perl script that will do the proper conversion for you:

./gpsdrive2gpx.pl track0000.sav > track0000.gpx

Those you can upload them to openstreetmap, at last.

Formatting numbers with iostream

What is the output of this C++ program?

#include <stdio.h>
#include <iostream>
#include <iomanip>

using namespace std;

main()
{
    printf("%05d\n", -3);
    cout << setw(5) << setfill('0') << -3 << endl;
}

It turns out that it is:

-0003
000-3

To get the numbers where you'd like them to be, you do it this way:

printf("%05d\n", -3);
cout << setw(5) << setfill('0') << internal -3 << endl;

Compare with C to see why C++ iostream is not really that much used. Also, setfill and internal are persistently changing the state of the stream, contributing fancy and interesting side effects:

cout << setw(5) << setfill('0') << internal << -3 << endl;
cout << setw(10) << "ciao" << endl;

this prints:

-0003
000000ciao

and I didn't see any sort of RAII save and restore of the stream state, nor a particularly obvious way to reset the stream to factory settings. Suppose that the function foo throws an exception here:

try {
    cout << setfill('=');
    cout << setw(10) << foo() << endl;
    cout << setfill(' ');
} catch (MyException& e) {
    // Catch this hypotetical exception
    cerr << setw(3) << e.errCode() << ": " << e.msg() << endl;
}

Then the state of the stream never gets reset, and the error code ends up being padded with equals instead of the default space. What would the alternative be, reset the stream state after each function call in the code? Or am I missing something?

Update: here are some suggested, but still not optimal, ways of dealing with it:

From Simon Richter:

std::cout << (std::stringstream() << "foo" << std::setw(8) << "bar").rdbuf() << std::endl;

From Peter De Wachter:

fstream tmp;
tmp.copyfmt(cout);
cout << setw(5) << setfill('0') << internal << -3 << endl;
cout.copyfmt(tmp);
cout << setw(10) << "ciao" << endl;

This is a possible RAII class to save the stream state (based on Peter De Watcher's approach), but I suspect that creating a fstream for every block that uses iomanip operators is rather heavyweight:

    struct SaveIOState
    {
            fstream tmp;
            ostream& s;
            SaveIOState(ostream& s) : s(s)
            {
                    tmp.copyfmt(s);
            }
            ~SaveIOState()
            {
                    s.copyfmt(tmp);
            }
    };

And this is a lightweight version, again thanks to Sylvain Sauvage:

class SaveIOState
{
    ios_base& s;
    ios_base::fmtflags f;
public:
    SaveIOState(ios_base& s) : s(s), f(s.flags())
    {}
    ~SaveIOState()
    {
        s.flags(f);
    }
};

How to work with python-m2crypto

This is a little howto on how to understand the m2crypto Python module, beyond the examples provided:

1. Build the documentation as explained here
2. Look at the documentation, which does not contain a single comment
3. Click the links that show the source (for example, of M2Crypto.SMIME.SMIME.verify)
4. Look what openssl function gets called by the function you are interested in (for example, pkcs7_verifysomething))
5. Google the name of that openssl function, trying a few variations, until you find the openssl manpage that tells you what it does, in detail (for example, pcks_verify)
6. Understand what it does, and backport the understanding to the thin convenience layer that m2crypto adds on top of it
7. Realise that now you also understand the corresponding openssl smime -verify commandline invocation, and that next time instead of reading the openssl manpage you should look at the openssl api docs instead.

Happy new year

A year ago we got in touch with various Taiwanese aboriginal tribes to try to start localisation efforts.

Thanks to the research the Taroko people did during 2007 and the prototype work of tonight, the Taroko people in Taiwan can see the computer calendar of the new year in their own language: