Entries for Planet Debian.

Upgrade Cyanogenmod with an encrypted phone

Cyanogenmod found an update, it downloaded it, then it rebooted to install it and nothing happened. It turns out that the update procedure cannot work if the zip file to install is in encrypted media, so a workaround is to move the zip into unencrypted external storage.

As far as I know, my Nexus 4 has no unencrypted external storage.

This is how I managed to upgrade it, I write it here so I can find it next time:

  1. enable USB debugging
  2. adb pull /cmupdater/cm-11-20141115-SNAPSHOT-M12-mako.zip
  3. adb reboot recovery
  4. choose "install zip from sideload"
  5. adb sideload cm-11-20141115-SNAPSHOT-M12-mako.zip
Posted Fri Dec 19 10:21:29 2014 Tags:

HTML5 Server-sent events

I have a Django view that runs a slow script server-side, and streams the script output to Javascript. This is the bit of code that runs the script and turns the output into a stream of events:

def stream_output(proc):
    '''
    Take a subprocess.Popen object and generate its output, line by line,
    annotated with "stdout" or "stderr". At process termination it generates
    one last element: ("result", return_code) with the return code of the
    process.
    '''
    fds = [proc.stdout, proc.stderr]
    bufs = [b"", b""]
    types = ["stdout", "stderr"]
    # Set both pipes as non-blocking
    for fd in fds:
        fcntl.fcntl(fd, fcntl.F_SETFL, os.O_NONBLOCK)
    # Multiplex stdout and stderr with different prefixes
    while len(fds) > 0:
        s = select.select(fds, (), ())
        for fd in s[0]:
            idx = fds.index(fd)
            buf = fd.read()
            if len(buf) == 0:
                fds.pop(idx)
                if len(bufs[idx]) != 0:
                    yield types[idx], bufs.pop(idx)
                types.pop(idx)
            else:
                bufs[idx] += buf
                lines = bufs[idx].split(b"\n")
                bufs[idx] = lines.pop()
                for l in lines:
                    yield types[idx], l
    res = proc.wait()
    yield "result", res

I used to just serialize its output and stream it to JavaScript, then monitor onreadystatechange on the XMLHttpRequest object browser-side, but then it started failing on Chrome, which won't trigger onreadystatechange until something like a kilobyte of data has been received.

I didn't want to stream a kilobyte of padding just to work-around this, so it was time to try out Server-sent events. See also this.

This is the Django view that sends the events:

class HookRun(View):
    def get(self, request):
        proc = run_script(request)
        def make_events():
            for evtype, data in utils.stream_output(proc):
                if evtype == "result":
                    yield "event: {}\ndata: {}\n\n".format(evtype, data)
                else:
                    yield "event: {}\ndata: {}\n\n".format(evtype, data.decode("utf-8", "replace"))

        return http.StreamingHttpResponse(make_events(), content_type='text/event-stream')

    @method_decorator(never_cache)
    def dispatch(self, *args, **kwargs):
        return super().dispatch(*args, **kwargs)

And this is the template that renders it:

{% extends "base.html" %}
{% load i18n %}

{% block head_resources %}
{{block.super}}
<style type="text/css">
.out {
    font-family: monospace;
    padding: 0;
    margin: 0;
}
.stdout {}
.stderr { color: red; }
.result {}
.ok { color: green; }
.ko { color: red; }
</style>
{# Polyfill for IE, typical... https://github.com/remy/polyfills/blob/master/EventSource.js #}
<script src="{{ STATIC_URL }}js/EventSource.js"></script>
<script type="text/javascript">
$(function() {
    // Manage spinners and other ajax-related feedback
    $(document).nav();
    $(document).nav("ajax_start");

    var out = $("#output");

    var event_source = new EventSource("{% url 'session_hookrun' name=name %}");
    event_source.addEventListener("open", function(e) {
      //console.log("EventSource open:", arguments);
    });
    event_source.addEventListener("stdout", function(e) {
      out.append($("<p>").attr("class", "out stdout").text(e.data));
    });
    event_source.addEventListener("stderr", function(e) {
      out.append($("<p>").attr("class", "out stderr").text(e.data));
    });
    event_source.addEventListener("result", function(e) {
      if (+e.data == 0)
          out.append($("<p>").attr("class", "result ok").text("{% trans 'Success' %}"));
      else
          out.append($("<p>").attr("class", "result ko").text("{% trans 'Script failed with code' %} " + e.data));
      event_source.close();
      $(document).nav("ajax_end");
    });
    event_source.addEventListener("error", function(e) {
      // There is an annoyance here: e does not contain any kind of error
      // message.
      out.append($("<p>").attr("class", "result ko").text("{% trans 'Error receiving script output from the server' %}"));
      console.error("EventSource error:", arguments);
      event_source.close();
      $(document).nav("ajax_end");
    });
});
</script>
{% endblock %}

{% block content %}

<h1>{% trans "Processing..." %}</h1>

<div id="output">
</div>

{% endblock %}

It's simple enough, it seems reasonably well supported besides needing a polyfill for IE and, astonishingly, it even works!

Posted Sun Dec 14 16:32:10 2014 Tags:

SSL "protection"

In my experience with my VPS, setting up pretty much any service exposed to the internet, even a simple thing to put a calendar in my phone requires an SSL certificate, which costs money, which needs to be given to some corporation or another.

When the only way to get protection from a threat is to give money to some big fish, I feel like I'm being forced to pay protection money.

I look forward to this.

Posted Thu Dec 11 15:35:25 2014 Tags:

Radicale and DAVDroid

radicale and DAVdroid appeal to me. Let's try to make the whole thing work.

A self-signed SSL certificate

Generating the certificate:

    openssl req -nodes -x509 -newkey rsa:2048 -keyout cal-key.pem -out cal-cert.pem -days 3650
    [...]
    Country Name (2 letter code) [AU]:IT
    State or Province Name (full name) [Some-State]:Bologna
    Locality Name (eg, city) []:
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:enricozini.org
    Organizational Unit Name (eg, section) []:
    Common Name (e.g. server FQDN or YOUR name) []:cal.enricozini.org
    Email Address []:postmaster@enricozini.org

Installing it on my phone:

    openssl x509 -in cal-cert.pem -outform DER -out cal-cert.crt
    adb push cal-cert.crt /mnt/sdcard/
    enrico --follow-instructions http://davdroid.bitfire.at/faq/entry/importing-a-certificate

Installing radicale in my VPS

An updated radicale package, with this patch to make it work with DAVDroid:

    apt-get source radicale
    # I reviewed 063f7de7a2c7c50de5fe3f8382358f9a1124fbb6
    git clone https://github.com/Kozea/Radicale.git
    Move the python code from git to the Debian source
    dch -v 0.10~enrico  "Pulled in the not yet released 0.10 work from upstream"
    debuild -us -uc -rfakeroot

Install the package:

    # dpkg -i python-radicale_0.10~enrico0-1_all.deb
    # dpkg -i radicale_0.10~enrico0-1_all.deb

Create a system user to run it:

    # adduser --system --disabled-password radicale

Configure it for mod_wsgi with auth done by Apache:

    # For brevity, this is my config file with comments removed

    [storage]
    # Storage backend
    # Value: filesystem | multifilesystem | database | custom
    type = filesystem

    # Folder for storing local collections, created if not present
    filesystem_folder = /var/lib/radicale/collections

    [logging]
    config = /etc/radicale/logging

Create the wsgi file to run it:

    # mkdir /srv/radicale
    # cat <<EOT > /srv/radicale/radicale.wsgi
    import radicale
    radicale.log.start()
    application = radicale.Application()
    EOT
    # chown radicale.radicale /srv/radicale/radicale.wsgi
    # chmod 0755 /srv/radicale/radicale.wsgi

Make radicale commit to git

    # apt-get install python-dulwich
    # cd /var/lib/radicale/collections
    # git init
    # chown radicale.radicale -R /var/lib/radicale/collections/.git

Apache configuration

Add a new site to apache:

    $ cat /etc/apache2/sites-available/cal.conf
    # For brevity, this is my config file with comments removed
    <IfModule mod_ssl.c>
    <VirtualHost *:443>
            ServerName cal.enricozini.org
            ServerAdmin enrico@enricozini.org

            Alias /robots.txt /srv/radicale/robots.txt
            Alias /favicon.ico /srv/radicale/favicon.ico

            WSGIDaemonProcess radicale user=radicale group=radicale threads=1 umask=0027 display-name=%{GROUP}
            WSGIProcessGroup radicale
            WSGIScriptAlias / /srv/radicale/radicale.wsgi

            <Directory /srv/radicale>
                    # WSGIProcessGroup radicale
                    # WSGIApplicationGroup radicale
                    # WSGIPassAuthorization On
                    AllowOverride None
                    Require all granted
            </Directory>

            <Location />
                    AuthType basic
                    AuthName "Enrico's Calendar"
                    AuthBasicProvider file
                    AuthUserFile /usr/local/etc/radicale/htpasswd
                    Require user enrico
            </Location>

            ErrorLog{APACHE_LOG_DIR}/cal-enricozini-org-error.log
            LogLevel warn

            CustomLog{APACHE_LOG_DIR}/cal-enricozini-org-access.log combined

            SSLEngine on
            SSLCertificateFile    /etc/ssl/certs/cal.pem
            SSLCertificateKeyFile /etc/ssl/private/cal.key
    </VirtualHost>
    </IfModule>

Then enable it:

    # a2ensite cal.conf
    # service apache2 reload

Create collections

DAVdroid seems to want to see existing collections on the server, so we create them:

    $ apt-get install cadaver
    $ cat <<EOT > /tmp/empty.ics
    BEGIN:VCALENDAR
    VERSION:2.0
    END:VCALENDAR
    EOT
    $ cat <<EOT > /tmp/empty.vcf
    BEGIN:VCARD
    VERSION:2.1
    END:VCARD
    EOT
    $ cadaver https://cal.enricozini.org
    WARNING: Untrusted server certificate presented for `cal.enricozini.org':
    [...]
    Do you wish to accept the certificate? (y/n) y
    Authentication required for Enrico's Calendar on server `cal.enricozini.org':
    Username: enrico
    Password: ****
    dav:/> cd enrico/contacts.vcf/
    dav:/> put /tmp/empty.vcf
    dav:/> cd ../calendar.ics/
    dav:/> put /tmp/empty.ics
    dav:/enrico/calendar.ics/> ^D
    Connection to `cal.enricozini.org' closed.

DAVdroid configuration

  1. Add a new DAVdroid sync account
  2. Use server/username configuration
  3. For server, use https:////
  4. Add username and password

It should work.

Related links

Posted Tue Dec 9 16:35:50 2014 Tags:

The smell of email

This was written in response to a message with a list of demotivating behaviours in email interactions, like fingerpointing, aggressiveness, resistance when being called out for misbehaving, public humiliation for mistakes, and so on

There are times when I stumble on an instance of the set of things that were mentioned, and I think "ok, today I feel like doing some paid work rather than working on Debian".

If another day I wake up deciding to enjoy working on Debian, which I greatly do, I try and make sure that I can focus on bits of Debian where I don't stumble on any instances of the set of things that were mentioned.

Then I stumble on Gregor's GDAC and I feel like I'd happily lose one day of pay right now, and have fun with Debian.

I feel like Debian is this big open kitchen populated by a lot of people:

  • some dump shit
  • some poke the shit with a stick, contributing to the spread of the smell
  • some carefully clean up the shit, which in the short term still contributes to the smell, but makes things better in the long term
  • some prepare and cook, making a nice smell of food and NOMs
  • some try out the food and tell us how good it was

I have fun cooking and tring out the food. I have fun being around people who cook and try out the food.

The fun in the kitchen seems to be correlated to several things, one of which is that it seems to be inversely proportional to the stink.

I find this metaphore interesting, and I will start thinking about the smell of a mailing list post. I expect it should put posts into perspective, I expect I will develop an instinct for it, so that I won't give a stinky post the same importance of a post that smells of food.

I also expect that the more I learn to tell the smell of food from the smell of shit, the more I can help cleaning it, and the more I can help telling people who repeatedly contribute to the stink to please try cooking instead, or failing that, just try and stay out of the kitchen.

Posted Fri Dec 5 11:51:49 2014 Tags:

C++11 talk examples

On 2014-11-27 I gave a talk about C++ and new features introduced with C++11: these are the examples. They are all licensed under the wtfpli version 2. See cxx11-talk-notes for the talk notes.

Note that the wrapper interfaces turns errors from the underlying libraries into exceptions, so the method calls just do what they should, without the need of documenting special return values for error messages, and removing the need for each library to implement yet another way of reporting errors.

Also note that all wrapper objects do RAII: you create them and they clean after themselves when they go out of scope.

The wrapper objects also have cast operators to make them behave as the pointer or handle that they are wrapping, so that they can be transparently passed to the underlying libraries.

(note: I had to add U+2063 INVISIBLE SEPARATOR to prevent noreturn statements to be misinterpreted by the blog formatter. If you copypaste the code and encounter issues, you may want to delete the noreturn statements and retype them)

A gcrypt hash class

This class is a light wrapper around gcrypt's hashing functions.

ezhash.h

#ifndef EZHASH_H
#define EZHASH_H

#include <string>
#include <gcrypt.h>

namespace ezhash {

class Hash
{
protected:
    // members can now be initialized just like this, without needing to repeat
    // their default assignment in every constructor
    gcry_md_hd_t handle = nullptr;

public:
    Hash(int algo, unsigned int flags=0);
    ~Hash();

    // Assign 'delete' to a method to tell the compiler not to generate it
    // automatically. In this case, we make the object non-copiable.
    Hash(const Hash&) = delete;
    Hash(const Hash&&) = delete;
    Hash& operator=(const Hash&) = delete;

    // Add a buffer to the hash
    void hash_buf(const std::string& buf);

    // Add the contents of a file to the hash
    void hash_file(int fd);

    // Get a string with the hexadecimal hash
    std::string read_hex(int algo=0);

    /// Pretend that we are a gcry_md_hd_t handle
    operator gcry_md_hd_t() { return handle; }
};

}

#endif

ezhash.cpp

#include "ezhash.h"
#include <unistd.h>
#include <errno.h>
#include <string>
#include <cstring>
#include <sstream>
#include <iomanip>
#include <stdexcept>

using namespace std;

namespace ezhash {

namespace {

// noreturn attribute, to tell the compiler that this function never returns
[[noreturn]] void throw_gcrypt_error(gcry_error_t err)
{
    string msg;
    msg += gcry_strsource(err);
    msg += "/";
    msg += gcry_strerror(err);
    throw runtime_error(msg);
}

string errno_str(int error)
{
    char buf[256];
#if (_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE
    strerror_r(errno, buf, 256);
    string res(buf);
#else
    string res(strerror_r(errno, buf, 256));
#endif
    return res;
}

[[noreturn]] void throw_libc_error(int error)
{
    throw runtime_error(errno_str(error));
}

}


Hash::Hash(int algo, unsigned int flags)
{
    gcry_error_t err = gcry_md_open(&handle, algo, flags);
    if (err) throw_gcrypt_error(err);
}

Hash::~Hash()
{
    gcry_md_close(handle);
}

void Hash::hash_buf(const std::string& buf)
{
    gcry_md_write(handle, buf.data(), buf.size());
}

void Hash::hash_file(int fd)
{
    char buf[4096];
    while (true)
    {
        ssize_t res = ::read(fd, buf, 4096);
        if (res < 0) ezfs::throw_libc_error();
        if (res == 0) break;
        gcry_md_write(handle, buf, res);
    }
}

std::string Hash::read_hex(int algo)
{
    unsigned char* res = gcry_md_read(handle, algo);

    unsigned int len = gcry_md_get_algo_dlen(
            algo == 0 ? gcry_md_get_algo(handle) : algo);

    // Format the hash into a hex digit
    stringstream hexbuf;
    hexbuf << hex << setfill('0');
    for (unsigned i = 0; i < len; ++i)
        hexbuf << setw(2) << (unsigned)res[i];

    return hexbuf.str();
}

}

Example usage

        ezhash::Hash sha256(GCRY_MD_SHA256);
        sha256.hash_buf("ciao\n");
        sha256.hash_buf("foo\n");
        cout << sha256.read_hex() << endl;

Simple sqlite bindings

Remarkably simple sqlite3 bindings based on lambda callbacks.

ezsqlite.h

#ifndef EZSQLITE_H
#define EZSQLITE_H

#include <sqlite3.h>
#include <string>
#include <functional>
#include <stdexcept>

namespace ezsqlite {

/// RAII wrapper around a sqlite3 database handle
class DB
{
protected:
    sqlite3* handle = nullptr;

public:
    // Open a connection to a SQLite database
    DB(const std::string& filename, int flags=SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE);
    DB(const DB&) = delete;
    DB(const DB&&) = delete;
    DB& operator=(const DB&) = delete;
    ~DB();

    /**
     * Execute a query, optionally calling 'callback' on every result row
     *
     * The arguments to callback are:
     *  1. number of columns
     *  2. text values of the columns
     *  3. names of the columns
     */
    // std::function can be used to wrap any callable thing in C++
    // see: http://en.cppreference.com/w/cpp/utility/functional/function
    void exec(const std::string& query, std::function<bool(int, char**, char**)> callback=nullptr);

    /// Pretend that we are a sqlite3 pointer
    operator sqlite3*() { return handle; }
};

}

#endif

ezsqlite.cpp

#include "ezsqlite.h"

namespace ezsqlite {

DB::DB(const std::string& filename, int flags)
{
    int res = sqlite3_open_v2(filename.c_str(), &handle, flags, nullptr);
    if (res != SQLITE_OK)
    {
        // From http://www.sqlite.org/c3ref/open.html
        // Whether or not an error occurs when it is opened, resources
        // associated with the database connection handle should be
        // released by passing it to sqlite3_close() when it is no longer
        // required.
        std::string errmsg(sqlite3_errmsg(handle));
        sqlite3_close(handle);
        throw std::runtime_error(errmsg);
    }
}

DB::~DB()
{
    sqlite3_close(handle);
}

namespace {

// Adapter to have sqlite3_exec call a std::function
int exec_callback(void* data, int columns, char** values, char** names)
{
    std::function<bool(int, char**, char**)> cb = *static_cast<std::function<bool(int, char**, char**)>*>(data);
    return cb(columns, values, names);
}

}

void DB::exec(const std::string& query, std::function<bool(int, char**, char**)> callback)
{
    char* errmsg;
    void* cb = callback ? &callback : nullptr;
    int res = sqlite3_exec(handle, query.c_str(), exec_callback, cb, &errmsg);
    if (res != SQLITE_OK && errmsg)
    {
        // http://www.sqlite.org/c3ref/exec.html
        //
        // If the 5th parameter to sqlite3_exec() is not NULL then any error
        // message is written into memory obtained from sqlite3_malloc() and
        // passed back through the 5th parameter. To avoid memory leaks, the
        // application should invoke sqlite3_free() on error message strings
        // returned through the 5th parameter of of sqlite3_exec() after the
        // error message string is no longer needed. 
        std::string msg(errmsg);
        sqlite3_free(errmsg);
        throw std::runtime_error(errmsg);
    }
}

}

Example usage

    // Connect to the database
    ezsqlite::DB db("erlug.sqlite");

    // Make sure we have a table
    db.exec(R"(
        CREATE TABLE IF NOT EXISTS files (
                name TEXT NOT NULL,
                sha256sum TEXT NOT NULL
        )
    )");

    // Read the list of files that we know
    map<string, string> files;
    db.exec("SELECT name, sha256sum FROM files", [&](int columns, char** vals, char** names) {
        if (columns != 2) return false;
        files.insert(make_pair(vals[0], vals[1]));
        return true;
    });

A fast Directory object

This is a lightweight wrapper around O_PATH file descriptors for directories. I'd love to see a library of well-maintained and thin C++ bindings around libc, that do little more than turning errors into exceptions and making it also work with std::string buffers.

ezfs.h

#ifndef EZFS_H
#define EZFS_H

#include <string>
#include <functional>
#include <memory>
#include <cerrno>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <dirent.h>

namespace ezfs {

class Directory
{
protected:
    int handle = -1;

public:
    Directory(const std::string& pathname, int flags=0);
    ~Directory();
    Directory(const Directory&) = delete;
    Directory(const Directory&&) = delete;
    Directory& operator=(const Directory&) = delete;

    /// List the directory contents
    void ls(std::function<void(const dirent&)> callback);

    int open(const std::string& relpath, int flags, mode_t mode=0777);
};

std::string errno_str(int error=errno);
[[noreturn]] void throw_libc_error(int error=errno);

}

#endif

ezfs.cpp

#include "ezfs.h"
#include <stdexcept>
#include <memory>
#include <cstring>
#include <cstdlib>
#include <string>
#include <linux/limits.h>

using namespace std;

namespace ezfs {

string errno_str(int error)
{
    char buf[256];
#if (_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE
    strerror_r(errno, buf, 256);
    string res(buf);
#else
    string res(strerror_r(errno, buf, 256));
#endif
    return res;
}

[[noreturn]] void throw_libc_error(int error)
{
    throw runtime_error(errno_str(error));
}

Directory::Directory(const std::string& pathname, int flags)
{
    handle = ::open(pathname.c_str(), O_PATH | O_DIRECTORY | flags);
    if (handle < 0) throw_libc_error();
}

Directory::~Directory()
{
    ::close(handle);
}

void Directory::ls(std::function<void(const dirent&)> callback)
{
    int fd = openat(handle, ".", O_DIRECTORY);
    if (fd < 0) throw_libc_error();

    // RAII Self-cleaning DIR object
    unique_ptr<DIR, std::function<void(DIR*)>> dir(fdopendir(fd), [](DIR* dir) { if (dir) closedir(dir); });
    if (!dir)
    {
        // fdopendir(3): After a successful call to fdopendir(), fd is used
        // internally by the implementation, and should not otherwise be used
        // by the application.
        //
        // but if the fdopendir call was not successful, fd is not managed by
        // DIR, and we still need to close it, otherwise we leak a file
        // descriptor.
        //
        // However, close() may modify errno, so we take note of the errno set
        // by fdopendir and raise the exception based on that.
        int fdopendir_errno = errno;
        close(fd);
        throw_libc_error(fdopendir_errno);
    }

    // Size the dirent buffer properly
    const unsigned len = offsetof(dirent, d_name) + PATH_MAX + 1;
    unique_ptr<dirent, std::function<void(void*)>> dirbuf((dirent*)malloc(len), free);

    while (true)
    {
        dirent* res;
        int err = readdir_r(dir.get(), dirbuf.get(), &res);

        // End of directory contents
        if (err == 0)
        {
            if (res)
                callback(*res);
            else
                break;
        } else
            throw_libc_error(err);
    }
}

int Directory::open(const std::string& relpath, int flags, mode_t mode)
{
    int res = openat(handle, relpath.c_str(), flags, mode);
    if (res < 0) throw_libc_error();
    return res;
}

}

Example usage

        // This is almost the equivalent of running "sha256sum ."
        ezfs::Directory dir(".");
        dir.ls([&](const dirent& d) {
            if (d.d_type != DT_REG) return;

            ezhash::Hash sha256(GCRY_MD_SHA256);
            // I have no RAII wrapper around file handles at the moment, so
            // I'll have to use a try/catch for cleaning up after errors
            int fd = dir.open(d.d_name, O_RDONLY);
            try {
                sha256.hash_file(fd);
                close(fd);
            } catch (...) {
                close(fd);
                throw;
            }

            cout << sha256.read_hex() << "  " << d.d_name << endl;
        });
Posted Sun Nov 30 19:26:26 2014 Tags:

C++11 talk notes

On 2014-11-27 I gave a talk about C++ and new features introduced with C++11: these are the talk notes. See cxx11-talk-examples for the examples.

(note: I had to add U+2063 INVISIBLE SEPARATOR to prevent noreturn statements to be misinterpreted by the blog formatter. If you copypaste the code and encounter issues, you may want to delete the noreturn statements and retype them)

Overview of programming languages

It has to be as fast as possible, so interpreted languages are out.

You don't want to micro manage memory, so C is out.

You don't want to require programmers to have a degree, so C++ is out.

You want fast startup and not depend on a big runtime, so Java is out.

[...]

(Bram Moolenaar)

C++ secret cultist protip

Do not call up what you cannot put down.

C++ is a compiled language

It is now possible to use the keyword constexpr to mark functions and objects that can be used at compile time:

/*
 * constexpr tells the compiler that a variable or function can be evaluated at
 * compile time.
 *
 * constexpr functions can also be run at run time, if they are called with
 * values not known at compile time.
 *
 * See http://en.cppreference.com/w/cpp/language/constexpr for more nice examples
 *
 * It can be used to avoid using constants in code, and using instead functions
 * for computing hardware bitfields or physical values, without losing in
 * efficiency.
 */

#include <iostream>

using namespace std;

constexpr int factorial(int n)
{
    return n <= 1 ? 1 : (n * factorial(n-1));
}

int main()
{
    cout << "Compile time factorial of 6: " << factorial(6) << endl;

    cout << "Enter a number: ";
    int a;
    cin >> a;

    cout << "Run time factorial of " << a << ": " << factorial(a) << endl;
}

See also this for more nice examples. See this and this for further discussion.

Multiline strings

        const char* code = R"--(
          printf("foo\tbar\n");
          return 0;
        )--";

See this.

C++ memory management protip

RAII: Resource Acquisition Is Instantiation

This is not new in C++11, but in my experience I have rarely seen it mentioned in C++ learning material, and it does make a huge difference in my code.

See this and this for details.

Constructors and member initializer lists

Initializers in curly braces now have their own type: std::initializer_list:

#include <string>
#include <iostream>
#include <unordered_set>

using namespace std;

// std::initializer_list<…>
//   will have as its value all the elements inside the curly braces

string join(initializer_list<string> strings)
{
    string res;
    for (auto str: strings)
    {
        if (!res.empty())
            res += ", ";
        res += str;
    }
    return res;
}

int main()
{
    unordered_set<string> blacklist{ ".", "..", ".git", ".gitignore" };

    cout << join({ "antani", "blinda" }) << endl;
}

See this for details, including the new uniform initialization trick of omitting parentesis in constructors so that you can call normal constructors and initializer_list constructors with the same syntax, which looks like an interesting thing when writing generic code in templates.

Type inference

I can now use auto instead of a type to let the compiler automatically compute the value of something I assign to:

        auto i = 3 + 2;

        // See also https://github.com/esseks/monicelli
        vector<string> names{ "antani", "blinda", "supercazzola" };
        for (auto i = names.cbegin(); i != names.cend(); ++i)
            cout << i;

        template<typename T>
        T frobble(const T& stuff)
        {
             // This will work whatever type is returned by stuff.read()
             auto i = stuff.read();
             // …
        }

See this for more details.

Range-based for loop

C++ now has an equivalent of the various foreach constructs found in several interpreted languages!

        for (auto i: list_of_stuff)
                cout << i << endl;

        for (auto n: {0,1,2,3,4,5})
                cout << n << endl;

        // This construct:
        for (auto i: stuff)

        // If stuff is an array, it becomes:
        for (i = stuff, i < stuff + sizeof(stuff) / sizeof(stuff[0]); ++i)

        // If stuff has .begin() and .end() methods it becomes:
        for (i = stuff.begin(); i != stuff.end(); ++i)

        // Otherwise it becomes:
        for (i = begin(stuff); i != end(stuff); ++i)

        // And you can define begin() and end() functions for any type you
        // want, at any time

See this and this for details.

Lambda functions and expressions

Lambdas! Closures!

Something like this:

// JavaScript
var add = function(a, b) { return a + b; }
# Python
add = lambda a, b: a + b

Becomes this:

auto add = [](int a, int b) { return a + b; }

And something like this:

// JavaScript
var a = 0;
$.each([1, 2, 3, 4], function(idx, el) { a += el });

Becomes this:

unsigned a = 0;
std::for_each({ 1, 2, 3, 4 }, [&a](int el) { return a += el; });

See this, this and this.

Tuple types

C++ now has a std::tuple type, that like in Python can be used to implement functions that return multiple values:

        tuple<int, string, vector<string>> parse_stuff()
        {
                return make_tuple(id, name, values);
        }

        string name; vector<string> values;

        // std::ignore can be used to throw away a result
        tie(ignore, name, values) = parse_stuff();

        // std::tie can also be used to do other kind of
        // multi-operations besides assignment:
        return tie(a, b, c) < tie(a1, b1, c1);
        // Is the same as:
        if (a != a1) return a < a1;
        if (b != b1) return b < b1;
        return c < c1;

See here, here and here.

Regular expressions

We now have regular expressions!

        std::regex re(R"((\w+)\s+(\w+))");
        string s("antani blinda");
        smatch res;

        if (regex_match(s, res, re))
            cout << "OK " << res[1] << " -- " << res[2] << endl;

The syntax is ECMAScript by default and can be optionally changed to basic, extended, awk, grep, or egrep.

See here and here.

General-purpose smart pointers

There is std::unique_ptr to code memory ownership explicitly, and std::shared_ptr as a reference counted pointer, and smart pointers can have custom destructors:

    unique_ptr<dirent, std::function<void(void*)>> dirbuf((dirent*)malloc(len), free);

See here and here.

Miscellaneous other cool things

Standard attribute specifiers

string errno_str(int error)
{
    char buf[256];
#if (_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE
    strerror_r(errno, buf, 256);
    string res(buf);
#else
    string res(strerror_r(errno, buf, 256));
#endif
    return res;
}

[[noreturn]] void throw_libc_error(int error)
{
    throw runtime_error(errno_str(error));
}

See here.

Hash tables

See here and look at the new containers unordered_set, unordered_map, unordered_multiset, and unordered_multimap.

Multithreading

There is a standard threading model, with quite a bit of library support: see here, here, here, and here for atomic data structures.

Variadic templates

Templates can now take variable number of arguments, and that opens possibilities for interesting code generation, like implementing a generic, type-safe printf statement, or something like this:

db.query(R"(
   INSERT INTO table NAMES (id, name, description)
     VALUES (?, ?, ?)
)", 4, "genio", "fantasia, intuizione, decisione, e velocità di esecuzione");

See here and here.

Essential tools

You need at least g++ 4.8 or clang 3.3 to have full C++11 support. They will be both available in jessie, and for wheezy you can use the nightly clang packages repository. I cannot think of a good excuse not to use -Wall on new code.

scan-build from clang is another nice resource for catching even more potential problems at compile time.

valgrind is a great tool for runtime code analysis: valgrind --tool=memcheck (the default) will check your program for wrong memory accesses and memory leaks. valgrind --tool=callgrind will trace function calls for profiling, to be analyzed with kcachegrind. valgrind --tool=helgrind can check multi-threaded programs for suspicious concurrent memory accesse patterns.

And of course gdb: a nice trick with C++ is to issue catch throw to get a breakpoint at the point where an exception is being thrown. help catch provides a list of other interesting catch examples.

Coredump tips: ulimit -c to enable core dumps, triggering a core dump with ^\, opening a core with gdb program core, and more details on man 5 core.

An extra gdb tip, which is not related to C++ but helped me considerably recently, is that it can be attached to running python programs to get a live Python traceback.

Posted Sun Nov 30 18:52:14 2014 Tags:

pdo-archive