Index of categories

Non-technical posts.

My semi serious stand up comedy notes

Video

Disclaimers

“Someone has said that it requires less mental effort to condemn than to think.”

(Emma Goldman, on several things including mailing list flamewars)

Fascinating Aïda's "Dogging" song.

Look for "dogging etiquette" for more examples of code of conducts. Just don't take your computer for repair immediately afterwards™.

Introduction

Every daring attempt to make a great change in existing conditions, every lofty vision of new possibilities for the human race, has been labeled Utopian.

(Emma Goldman, on the Debian Social Contract)

I am going to talk about many topics that we all know have so much in common:

  • Anarchism
  • Poliamory
  • BDSM
  • and Free Software

They are all, after all:

  • People
  • Consensually
  • Doing Things
  • Together

BDSM

A person is no less a slave because they are allowed to choose a new master once in a term of years.

(Lysander Spooner about proprietary cloud service providers)

If you thought you've seen it all with recursive acronyms, here's a chain acronym: Bondage Discipline, Dominance Submission, Sado Masochism.

Why I think BDSM is interesting: not (just) because of whips, but for having a lot of awareness about power releationships. Why should one accept from a coworker a level of abuse that would be considered a hard limit when negotiating with a trusted dom?

The BDSM Free Software definition: "I refuse to be bound by software I cannot negotiate with".

YKINMKBYKIOK (Your Kink Is Not My Kink But Your Kink Is Okay) is a nice example of dealing with diversity, and it also definitely solves the emacs vs vi debate.

Comfort zones, safewords, traffic light flow control, safety.

"No means no", unless there has been a long discussion first, and a safeword is in place, in which case "Fuzzy purple unicorn" means "no"

"No means no", and if someone insists after a "no", it becomes harassment.

"No means no" is a precondition for being able to say "yes": http://pervocracy.blogspot.de/2011/03/no-and-no-and-no-and-yes.html

Aftercare! Aftercare! Release parties! High fives! Solidarity after flamewars or votes!

Poliamory

If love does not know how to give and take without restrictions, it is not love, but a transaction that never fails to lay stress on a plus and a minus.

(Emma Goldman, on volunteer projects)

Polyamory is the practice, desire, or acceptance of intimate relationships that are not exclusive with respect to other sexual or intimate relationships, with knowledge and consent of everyone involved.

Compersion, n: the feeling you get when someone else also takes good care of one of your packages.

We currently allow only one value in the Maintainer field: * takeover is traumatic, because values can only be replaced * if values could be added instead, and removed when they don't make sense anymore...

What is your definition of love? My current one is: my world is better with you in it.

Relationship anarchy is the practice of forming relationships which are not bound by rules aside from what the people involved mutually agree on. How do you call a relationship that is bound by rules that the people involved do not agree on?

From discussions after the talk

New Relationship Energy, the excitement when you start to maintain a new package, and the risk of been carried away by the excitement and neglecting all the other ones.

Consent

Anarchism, to me, means not only the denial of authority, not only a new economy, but a revision of the principles of morality. It means the development of the individual as well as the assertion of the individual. It means self-responsibility, and not leader worship.

(Voltairine de Cleyre about trusting lintian warnings)

You need to know what you are doing, and what situation you're putting yourself into.

You need to know that the person asking a question really is able to accept any answer, and take it seriously.

You need to feel that you have alternatives.

Be selfish when you ask, honest when you reply, and when others reply, take them seriously. If any of this doesn't stand, I find it hard to trust that we are in a consensual situation.

When is one supposed to learn about consent?

  • I see little consensuality in standard education.
  • I see little consensuality at work.

Consent explained with tea.

Practical advice

Anarchism has but one infallible, unchangeable motto, ‘Freedom.’ Freedom to discover any truth, freedom to develop, to live naturally and fully.

(Lucy Parsons about the DFSG)

Relationship advice and work advice have a lot in common:

Relationship advice from 99 ways to ruin an open source project

Online participation advice from How to Screw Up Your Relationship (and make everyone miserable while you’re at it)

Packaging advice from BDSM Basics: 20 Unsolicited Tips for New Dominants

Advice about joining a new community from Advice to a newbie submissive about dominants

♥ ♥ ♥

Dear Debian, and dear everyone contributing to it: my world is better with you in it.

I love you all :* <3

Posted Tue Aug 18 11:48:08 2015 Tags:

Expectations and needs

All people ever say is: "thank you" (a celebration of life) and "please" (an opportunity to make life more wonderful). (Marshall Rosenberg)

Sometimes, when I see the word "expectation" I try to read it as "need" and see how things change.

I noticed that this tends to reframe situations in a way that makes me feel more comfortable.

I noticed that I tend to instinctively perceive "expectations" as "do this or there will be consequences", and I tend to instinctively perceive "needs" as "do this if you want to see me happy".

I noticed that my motivation to care for someone's expectations tend to be something close to fear, and my motivation to care for someone's needs tends to be something close to love.

This might give me a bit more hints on The art of asking: I will not expect you to do something for me, I'll just allow myself to be loved, liked or helped by you, and I'll try to be open about what I need.

I smile realising that since a long time, on the professional side of my life, I learnt to lead interaction with my customers along the same lines: "let's talk about what you need, not about what you expect of me".

Posted Fri Aug 14 12:13:16 2015 Tags:

Random quote

Be selfish when you ask, honest when you reply, and when others reply, take them seriously.

(me, late at night)

Posted Sun Jul 19 18:53:03 2015 Tags:

Love thy neighbor as thyself

‘Love thy neighbor as thyself’, words which astoundingly occur already in the Old Testament.

One can love one’s neighbor less than one loves oneself; one is then the egoist, the racketeer, the capitalist, the bourgeois. and although one may accumulate money and power one does not of necessity have a joyful heart, and the best and most attractive pleasures of the soul are blocked.

Or one can love one’s neighbor more than oneself—then one is a poor devil, full of inferiority complexes, with a longing to love everything and still full of hate and torment towards oneself, living in a hell of which one lays the fire every day anew.

But the equilibrium of love, the capacity to love without being indebted to anyone, is the love of oneself which is not taken away from any other, this love of one’s neighbor which does no harm to the self.

(From Herman Hesse, "My Belief")

I always have a hard time finding this quote on the Internet. Let's fix that.

Posted Wed May 20 11:35:15 2015 Tags:

Free as in Facebook

Yesterday we were in an airport. We tried to connect to the airport "free" wifi. It had a captive portal that asked for a lot of personal information before one could maybe get on the internet, and we gave up. Bologna Airport, no matter what they do to pretend that they like you, it's always clear that they don't.

I looked at the captive portal screen and I said: «ah yes, "free" wifi. Free as in Facebook».

We figured that we had an expression that will want to be reused.

Posted Mon Mar 9 10:58:49 2015 Tags:

<3 Bremen

Greedy people exploit our cultural differences to justify fighting wars to seize wealth for themselves, when sharing and enjoying cultural diversity makes us far richer than if we possess material wealth

Greedy people exploit our cultural differences to justify fighting wars to seize wealth for themselves, when sharing and enjoying cultural diversity makes us far richer than if we possess material wealth!

From a recursive murales in Bremen Halmerweg.

Posted Sun Jan 11 22:20:11 2015 Tags:

Non importa che mi dai del voi

Dai, non importa che mi dai del voi

In che senso?

Eh, mi dici sempre "voi informatici", "voi tecnici", "voi..."

Posted Fri Dec 19 15:55:20 2014 Tags:

On relationships

Good relationships are like a good video game

with an easy, intuitive interface

and lots of interesting content.

(Lynoure)

Posted Wed Aug 6 16:47:40 2014 Tags:

On love and sexual desire

Soundtrack: Skullcrusher Mountain (with lyrics)

After seeing a review of it, I just watched again the first episode of Lupin III. with an eye on how sex and love are represented. To my eyes, they seem to be shown as mutually exclusive: the evil boss sexual appetites have Fujiko tied up and (childishly) "raped". Lupin's love of Fujiko is shown as self-sacrifice, with him ending up in Zenigata's cuffs, with him enduring her betrayal. In the end, Lupin's sexual desire for Fujiko seems to turn her again into a disadvantaged position, as the instrument of "rape" of the evil boss comes back into play.

In all the literature that I remember as I grew up, there has been love and there has been sexual desire. Love was wishing for the other person to be well, sexual desire was wishing for oneself to be well.

For some reason, in most of the literature there seemed to be an implicit rule saying that two people cannot both be well at the same time. Love meant being soulless and devoting oneself to the other person entirely, and suffering from it. Sexual desire meant reducing the other person as an object of one's own pleasure, and (usually she) would suffer from it.

Sometimes two people loved each other and wished for both to be well at the same time, and were usually torn away by circumstances, or had to suffer and sacrifice a lot so that they could be together, or one of them would die, for extra drama. What being together meant, was usually not covered in the story, and tended to happen during that "happily ever after" that usually starts where the book or film ends.

Yet, I find that sex is a wonderful way for two people to be both happy at the same time, where I can desire my happiness and the other person's happiness, and where each feeling, each desire, each move contributes to both. In pleasing myself I please the other; in pleasing the other I please myself.

Is there a romantic story where the lovers do not just feel love, but also they long for, they desire each other? Where that desire is shown not by hitting the partner in the head with a club and dragging them to wake up tied up in a secret lair, but by inviting their partner to come closer, kissing them, holding them tight to their body, caressing them, discovering what gives them pleasure, opening up for them to play, each person riding their own, the other's, and their shared desire?

I cannot think of one. It sounds like I'll have to write one.

With my own life.

Posted Tue Jul 22 17:40:36 2014 Tags:

Abuse

I've recently spent a lot of effort trying to find, recover, reconnect, embrace, strengthen and grow my inner child, my actual identity, what I'm comfortable being. Now I figure that I feel abused when I perceive an assault against my identity.

I'm ok having had to study a shallow moralistic piece of literature at school. Having been expected to like it and embrace it, that feels like abuse.

I'm ok being told I made a mistake. Being humiliated for it, feels like abuse.

I'm ok being suggested how to avoid making a mistake. But being expected to promptly follow the advice and change my life accordingly, that feels like abuse.

Someone offers help in cleaning my house? Fine. Someone barges in, takes over my personal space and reshapes it as they see fit? That feels like abuse.

If I'm sad, a hug's great. If the hug comes with a surprise grope at my bottom, that feels like abuse. I would not be bothered about the sexual assault, but about the denial of my current state of mind. The denial of feelings. Forget who you are, how you feel, and just be my sexual object. When it begins to deny my identity, then it starts feeling like abuse.

Being taught that "when you love a person, you must..." feels like abuse.

Being expected to have sex and like it at another person's whim, regardless of my real feelings, feels like abuse.

Being expected to like to have sex, or not like to have sex, regardless of my real feelings, feels like abuse.

The concept of "marital duty" feels like abuse. Two people who are expected to demand and provide sex, just because they are married, regardless of what they actually feel or need.

The concept of "training your boyfriend" sounds to me like nonconsensual manipulation. Like Abuse.

A lot of dating advice? Abuse. Abuse. Abuse.

A relationship based on me faking my identity, in order to keep another person close to me who wouldn't be otherwise, sounds to me like a relationship based on mutual abuse.

Why do we even have the concept of love potions in our culture, in our fairy tales?

"For your own good", abuse. Luckily, there are alternatives.

Being called "nice boobies" when you want to be called "Elizabeth", feels to me like abuse.

Being called "geek" when you want to be called "Enrico", feels to me like abuse.

Being called a stereotype when I want to be recognised as an individual, feels like abuse. When I grow up in an environment where that is the accepted way of addressing people, and so I learn to do the same, then I feel abused and I'm taught to abuse.

I think I had some pretty abusive role models when I grew up.

Being asked to do something I don't want to? Fine. Telling me off because I do not like it? Abuse.

Despising me because I like something another person doesn't? Abuse.

Despising me because I don't like something another person likes? Abuse.

As a kid I quickly had to learn to figure out what a person liked or disliked, and pretend accordingly. To this day, I still have a problem answering the simple question "what would you like to do today?"

It feels like abuse when my identity is denied. When I cannot make mistakes. When I cannot be vulnerable. When I have to be happy. When I have to be sad. When I have to care. When I must not care. When it doesn't matter how I feel, who I am, but I just have to feel something, like something, be something.

For my own good. Because someone else knows better. Because "that's the thing you do". Because.

When I was a kid, I felt very uncomfortable when going to a carnival or to a theme park, because I felt an expectation to enjoy it, regardles of what I was really feeling. An effort was made to make me happy so I had to. It was a place where I was not free to have my own mood. Abuse.

The doctor would touch, knock, hit, tell me to do this or that, but would never ask me how I was feeling. He would tell me how I was feeling, and he must have been right. Abuse.

There was lots of abuse against my identity in my growing up. I had to learn to blend in to protect myself. I had to learn to please. I had to pretend I liked it all. I got good at pretending.

In order to be accepted as a person I needed to pretend to be someone else. And the person that ended up being accepted was not me.

In "A Wizard from Earthsea", Ged turns into a falcon to run away, but stays a falcon for too long, and forgets how to be a person again. The wizard Ogion turns him back by recognising him, accepting him, and speaking only one word: Ged's name.

I feel abused when I'm taken away from my identity, my feelings, my needs. I could even do that to myself, out of frustration, out of despair, out of habit.

I have started to recognise the moments when I'm not feeling abused, the people who accept me for what I am, there and then, who allow me to exist without judging me. I have started to accept all other moments as likely abuse attempts, and emotionally deal with them accordingly.

I have started to become conscious of when I'm abusing myself (this has been an interesting read), and stop, and ask myself why.

Posted Sat Jun 21 14:38:12 2014 Tags:

Fear of losing

If I am afraid of breaking my laptop, then I may leave it at home, and it will be as if I didn't have a laptop.

If I am afraid of losing faith, then I may closely follow the dictates of the church. I will be keeping the church's faith, but not mine.

If I am afraid of losing my children, they may have to run away from me to be free to grow into adults.

If I am afraid of losing my inner child, then I might not expose it to the world, and so my inner child will never live. I will just be a box, a mask of what the world expects from me, that shelters and cages the Me that would like to live.

If I am afraid of losing you, then I may get obsessed with preserving the beautiful image I have of you. I may stop seeing, experiencing you as you live, think, grow. I may become afraid of your depth, of your being different each day, of your being alive. I may end up with cherishing a perfect image of you in my head, while you will have become a stranger to me.

I am really asking when I can accept answers.

I am really living when I can accept myself.

I am really loving when I can accept you.

Posted Mon Jun 16 09:21:52 2014 Tags:

Perfection

We like perfection.

Perfection is the ultimate achievement, there is nothing beyond.

Perfection is fully understood. It is not going to change, it is fact, we can rely upon it.

Perfection is final. Perfection is death.

Ideas can be perfect, and perfect ideas are easy to understand.

Perfect ideas are final and unchangeable. Perfect ideas are hard to correct, hard to refute.

Perfect ideas spread easily. They are helpful. They shed light on a little corner of our world, give it shape. They bring stability. They can be relied upon. Perfect ideas make good memes.

Perfect ideas are shared standards through which we act, interact, coordinate, cooperate. They don't change, so they are a solid base for habits, that make a bit of our life a little easier.

That we should not kill, is a perfect idea. So is racism. So are the ten commandments, so, for many, is love.

Thanks Lynoure for saying the right thing at the right time.

Posted Wed Jun 4 22:53:37 2014 Tags:

When I said "I love you"

All people ever say is: thank you (a celebration of life) and please (an opportunity to make life more wonderful). Marshall Rosenberg

I have said "I love you" many times in my life, and many times I have failed to say it, because, for me, it is not an easy thing to say.

It is not easy when I have no idea what the other person will make of it: will they be frightened? Will they feel awkward around me afterwards? Will they disappear from my life?

But do I know what I myself mean when I say it?

I have said "I love you" because I thought you somehow expected it of me. "please, consider me worth of you".

I have said "I love you" to beg for affection. "please, love me back".

I have said "I love you" because I was grateful to you for existing in my life. "thank you".

I now understand why it has not been easy for me to say "I love you" when I was feeling, or imagining, that I had to say it.

I now understand why I have sometimes made myself awkward, as I was begging.

I now understand why, when I said "I love you" out of gratitude, when I said it to celebrate that you exist in my life, that's when I felt no trouble, no fear, and when I felt that my words really were fitting with what I was feeling and what I was wanting to say.

Posted Mon Jun 2 10:18:56 2014 Tags:

Habits

Beware of habits. I've seen them turn into expectations over time, without any kind of negotiation.

Posted Thu Feb 13 01:22:59 2014 Tags:

Original sin

I feel that I was somewhat born innocent, an animal with sound, primal instincts. But if I remained that way, I wouldn't be able to function in a complex society, so I got education. Education taught me what is expected from me in order to be accepted by my peers[1].

Education taught me more than that: it also taught me to enjoy what a complex society can give me: art, science, history, philosophy, adding depth of meaning and correlations to my perceptions and memories.

Education wasn't perfect, though. Some of my educators obsessed about some of the expectations, and gave me rules to follow that aren't really needed to interact with a society. Sit down for hours in silence without complaining. Don't talk back to figures of authority, even when they are abusing me. Don't ever feel that my efforts in a task have been enough, because there is always something more that I could have done. Do what people expect me to do, regardless of what I wish to do. Do what I need to do, not what I want to do.

So I grew up with a set of arbitrary expectations that weren't needed to function in a society, and weren't in any way meeting any of my needs, yet I still felt them as a part of me, putting every effort into meeting them that I would put in making myself happy. I like to call this set of learned arbitrary, unneeded expectations "neurosis".

I like this as an interpretation of the "original sin" myth: in order to go from an innocent animal to a member of a complex society, I acquired a set of neuroses that make me behave in a meaningless way. Or, rephrased going along with the myth, that rob me of my innocence.

In some environments like BDSM, many have a name for an unnegotiated practice forced on a person: they call it "abuse". It made sense to accept that I have been abused many times while being educated, because my educators also had their own baggage of parasite expectations, of neuroses. And since I don't currently know how I can ever be sure that I freed myself from all my neuroses, I feel like I should accept that it is possible that I do and will abuse others; by accepting it as a possibility, I hope at least to be able to realise as soon as possible that I am doing it, and try to stop.

I have now realised that my life is far simpler and more rewarding when expectations are negotiated between all the parties involved. I have recently spent a substantial amount of my energy in recognising and renegotiating many of the expectations and the neuroses that I had learnt while growing up, and, who knows, perhaps this effort will continue throughout my life.

So there you go, an agnostic atheist freeing himself of his original sin, and regaining his paradise lost one step at a time, alive, on earth.

And enjoying every moment of it.

[1] I suspect that is why people educated in a high class society tend to be accepted more easily by a high class society: they conform to the right set of expectations. But I digress.

Posted Sat Feb 8 17:48:27 2014 Tags:

On political correctness

I am reading "Four ways to forgiveness" by Ursula Le Guin. This is how the book begins:

“On the planet 0 there has not been a war for five thousand years,” she read, “and on Gethen there has never been a war.” She stopped reading, to rest her eyes and because she was trying to train herself to read slowly, not gobble words down in chunks the way Tikuli gulped his food. “There has never been a war”: in her mind the words stood clear and bright, surrounded by and sinking into an infinite, dark, soft incredulity.

What would that world be, a world without war? It would be the real world Peace was the true life, the life of working and learning and bringing up children to work and learn. War, which devoured work, learning, and children, was the denial of reality. But my people, she thought, know only how to deny. Born in the dark shadow of power misused, we set peace outside our world, a guiding and unattainable light. All we know to do is fight. Any peace one of us can make in our life is only a denial that the war is going on, a shadow of the shadow, a doubled unbelief.

That made me realise that several times I perceived political correctness as "only a denial that the war is going on".

Last night I watched this video, which I felt was very relevant to this.

Indeed, I have more respect for someone who listens to me and makes a good effort to understand what I said, then rudely disagrees, than for someone who is very politely ignoring everything I am trying to say.

This is how I feel like this experience can be distilled into an actionable item:

Replying to an email is probably going to be useless, unless I am willing to make an effort to understand what the other person is trying to say, from their own point of view.

Posted Wed Jan 29 10:44:14 2014 Tags:

Liberation vs expectations

I feel like I found a key struggle: liberation vs expectations.

Many ideas liberate me, like poliamory, atheism, anarchism, nonviolent communication, sexual liberation, discordianism, free software, nonjudgemental environments, E-prime, sillyness, love, pataphysics, dada, constructivism, rolling naked in the mud, fantasy, bdsm exploration, pansexuality, this very idea that i'm trying to describe, crying, laughing, mindfulness, what have you.

Sometimes however they become an end rather than a mean, they give me expectations and judgements, and take some other freedom away.

I feel like I found where to draw a line: it's the point inbetween freeing myself of my limits and making myself some new ones.

It's one of those epiphany moments. Weeeeeee!

Posted Sun Jan 12 12:17:39 2014 Tags:

Spelling a chilometri zero

Lo spelling internazionale è troppo globalizzato, e volete recuperare un attimo la dimensione del posto dove siete nati e cresciuti?

Da oggi c'è questo script che fa per voi: gli dite dove abitate, e lui vi crea lo spelling a chilometri zero.

$ git clone git@gitorious.org:trespolo/osmspell.git
$ cd osmspell
$ ./osmspell "San Giorgio di Piano"
1: San Giorgio di Piano, BO, EMR, Italia
2: San Giorgio di Piano, Via Codronchi, San Giorgio di Piano, BO, EMR, Italia
3: San Giorgio Di Piano, Via Libertà, San Giorgio di Piano, BO, EMR, Italia
Choose one: 1
Center: 44.6465332, 11.3790398
A Argelato, Altedo
B Bentivoglio, Bologna, Boschi
C Cinquanta, Castagnolo Minore, Castel Maggiore, Cento
D Dosso
E Eremo di Tizzano
F Funo di Argelato, Finale Emilia, Ferrara, Fiesso
G Gherghenzano, Galliera, Gesso
I Il Cucco, Irnerio, Idice
L Località Fortuna, Lovoleto, Lippo
M Malacappa, Massumatico, Minerbio, Marano
N Navile
O Osteriola, Ozzano dell'Emilia, Oca
P Piombino, Padulle, Poggio Renatico, Piave
Q Quarto Inferiore, Quattrina
R Rubizzano, Renazzo, Riale
S San Giorgio di Piano, Saletto
T Torre Verde, Tintoria, Tombe
U Uccellino
V Venezzano Mascarino, Vigarano Mainarda, Veduro
X XII Morelli
Z Zenerigolo, Zola Predosa

I dati vengono da OSM, e lo script è un ottimo esempio di come usarne la API di geolocazione (veloci) e la API di query geografica (lenta).

Posted Sat Jan 4 00:38:16 2014 Tags:

Poesia: "Lavatrice"

Pensavo fosse pail,

invece ora è feltro.

Posted Tue Dec 3 22:32:23 2013 Tags:

Shops

Christmas songs should only ever be played on Christmas day.

In church.

At midnight.

Unless I happen to be there.

Posted Mon Dec 2 14:07:58 2013 Tags:

Explanation of umarell

Umarell /uma'rɛl/ (oo-mah-rell), n; pl. Umarells. People in a community who offer all sorts of comments to those who are trying to get some work done, but who are not doing any work themselves.

Etymology and further details

Umarell is a word that entered Italian slang in Bologna and is spreading to nearby towns, occasionally even across Italy. It comes from the Bolognese for "cute/odd little man".

"Umarells" are those people, usually retired men, who spend time watching construction works, often holding their hands behind their back, occasionally commenting on what is going on, sometimes trying to tell the workers what to do.

It's easy to find examples on the internet; the word was popularised by a blog collecting photos, which has even been published into a book.

With some Italian Debian friends, we realised that umarell is the perfect word to describe those people in a community, who offer all sorts of comments to those who are trying to get some work done, but who are not doing any work themselves.

I think that it is a word that fits perfectly, and since I'm likely going to use it blissfully anywhere, here is a page that temporarily explains what it means until the Oxford English Dictionary picks it up.

Posted Fri Sep 20 13:27:07 2013 Tags:

Random notes from that other lightning talk session

If you can, have dinner with your upstreams at least once a year.

Posted Sat Aug 31 12:35:09 2013 Tags:

On codes of conduct

A criticism of the status quo, with a simple proposal: video and lyrics.

With compassion unyielding to grudge, mother, I learnt how to love.

Posted Sat Aug 24 18:15:31 2013 Tags:

Random notes from that other lightning talk session

YKINMKBYKIOK: "Your Kink Is Not My Kink But Your Kink Is Ok"

As far as I'm concerned, this puts the vim vs emacs quarrel to rest, for good.

Posted Fri Aug 23 18:11:30 2013 Tags:

Random notes from that other lightning talk session

The BDSM Free Software Definition:

I refuse to be bound by software I cannot trust and negotiate with.

Posted Wed Aug 21 08:16:15 2013 Tags:

Random notes from DebConf

Compersion, n: the feeling you get when someone else also takes good care of one of your packages.

Posted Tue Aug 20 17:44:52 2013 Tags:

Components in a system

But what we are discovering is that if we see ourselves as components in a system,

that it is very difficult to change the world.

It is a very good way of organising things, even rebellions,

but it offers no ideas about what comes next.

(from All Watched Over By Machines of Loving Grace)

Posted Wed Mar 13 01:30:34 2013 Tags: tags/quotes

Che coss'è l'amor

Di recente mi sono spesso trovato a chiedermi "che cos'è l'amore", e ogni volta inizia a suonarmi in testa: "♪...chiedilo al vento / che sferza il suo lamento sulla ghiaia del viale del tramonto... ♫"

Socialmente l'amore mi è sempre stato venduto come una cosa indescrivibile ma allo stesso tempo estremamente codificata. L'amore è quella cosa che ti fa sentire così, e poi se non fai così non è vero amore, e l'altra persona la devi trattare così, e devi essere spontaneo mapperò... tv e musica con i quali volente o nolente sono cresciuto hanno spesso propagandato modelli che raramente ho condiviso, e che riassumerei con questo.

Mi è sempre stato difficile capire quando poter dire "ti amo". Il problema non è tanto nell'interpretare i miei sentimenti, quanto nel capire quali significati sto tirando addosso a entrambi nel momento in cui lo faccio. Per questo motivo, finché non sono sicuro di come l'espressione "ti amo" significhi per entrambi, tendo a ragionare usando invece concetti definiti in maniera un po' più chiara (per me), tipo: desiderio, fiducia, intimità, odori, intesa, ridere, contatto fisico, curiosità...

È un peccato: ogni tanto provo quel sentimento di piacevole resa verso l'altra persona che mi porta a voler dire "ti amo!", e finisco per non farlo.

Detto questo, da quando ho visto "Una de zombis", alla domanda "ma sei innamorato?" non può non venirmi in mente questo.

Posted Wed Feb 20 02:02:30 2013 Tags:

On praising people, and on success

This morning I was pointing out to friends how excellent is mako's post on Aaron Swartz, and I thought it'd be nice if we didn't have to wait for people to die before telling the world how awesome and inspirational they are.

Then Russ posted an article about work, success and motivation and I went to tell my friends how awesome and inspirational he is.

I, too, see myself as somehow successful, and I, too, don't identify in the usual stereotype of success. I don't want to stop being a craftsman to become a manager, I don't get a high from having power over other people, I don't define my value in terms of my profits.

At a glance, people don't see me as successful, until they get to know me better. They they realise that I'm not at all unhappy about my life.

I have a job that I like, I write Free Software and it gets used and appreciated, my colleagues are friends, who respect me and my opinion, and I respect them and theirs.

I can work from home. In fact, I can work from everywhere as long as I have my laptop with me. I can sustain a long distance relationship because I can work from the house of my partner when I'm visiting. Two days ago I worked from the bar of a farm on top of a hill, because I was on the road, it was close by, and what the hell, it's a wonderful place to be.

To me success means that I can care about the quality of my life, that I have the luxury of caring about little things that make my day, of trying to make good ideas sustainable, of working a bit more when I'm on fire, and of working a bit less when there's something wonderful in the world to see, or someone interesting in the world to meet.

Russ, the way I read your article, you are questioning what "success" means, and you are spot on. People should be able to define "success" as whatever works for them and pursue it freely. Only then success becomes something that is worth praising when it is achieved. Only then it becomes inspirational.

I like how you managed to put into words something that has been for a long time in some corner of my mind and I hadn't yet managed or bothered to bring into the spotlight.

You have the insight and the confidence of seeing something in an insightful but non-mainstream way, and say "you know what? That actually makes sense."

Sometimes I read your post, nod a lot and realise how important something actually is, how that is actually such an important part of myself. And now that you took it out for me to see it, I can appreciate how valuable it is, and make sure I don't accidentally lose it.

Thanks! That's another one I owe you. It's just the kind of thing I shouldn't wait before letting you know.

Posted Fri Jan 25 12:45:49 2013 Tags:

Yet another Ubuntu anecdote

Some posts on planet made me remember of a little Canonical-related story of mine.

Many years ago I shortly contracted for Canonical. It was interesting and fun.

At the time I didn't have any experience of being temporarily hired by a foreign company, so I rang my labour union to get an appointment, to make sure with them that everything was allright.

The phone call went more or less like this:

Me:

Hello. I have received this contract for temporary employment by a foreign company and I wondered if I could book an appointment to come show it to you to see if it's all ok.

Their answer rather cut me short:

Hi. Be careful! People get temporary employment from obscure companies with the headquarters, like, in the Isle of Man, they do the job, the company disappears and they never get paid. There's bad stuff out there!

I looked at the contract, the heading said something like "Canonical ltd, Douglas, Isle of Man".

I was certain that the union people would have never understood what was going on. I politely thanked them for their time and hung up. However, to this day I still regret that I didn't insist:

Uh, yes, the company is indeed in the Isle of Man. But what if I told you that it's owned by an astronaut?

I just signed the contract and had a good time.

Posted Sat Jan 15 10:35:36 2011 Tags:

La paura e la voglia

Results 1 - 10 of about 5,470 for "la paura e la voglia di essere nudi". (0.34 seconds) 

Results 1 - 10 of about 26,500 for "la paura e la voglia di essere soli". (0.10 seconds) 

Poi non c'è da stupirsi se siamo diventati come siamo diventati.

Posted Wed May 5 13:36:24 2010 Tags:

Se è vero questo...

Se è vero questo...

...allora è vero anche questo!

Posted Thu Apr 22 09:26:08 2010 Tags:

Global trends

For some time I have been trying to pinpoint what it is that is brewing in Italy and risks spreading elsewhere, like it happened in the past.

I don't need to be decent to stay in power

While following a train of thought during a political/philosophical lecture I figured that a current growing trend is to have public figures that are more and more indecent.

In Italy it is very hard to find a public figure you can look up to. It is hard to name a politician that is not involved in some shady exchange of favours or some abues of their powers, and we got used to seeing people in power implicated in major corruption scandals, perverted prostitution affairs, or dealings with international criminal organisations.

They do not normally end up in jail, and in fact they keep being very firmly in power, because they manage to stretch or change the laws to get away, or at least to delay trials in order to trigger some statute of limitations.

Is there a pattern here that, although maybe not as clearly defined as in Italy, can be found more or less globally?

Yesterday I thought that this could be such a pattern:

I don't need to be decent to stay in power

If I think of it like that, then it is most definitely not just an Italian phenomenon. If you tell "one doesn't need to be decent to stay in power" to a British, or to a French, I would not expect them to see anything strange with it. We all find it depressing, but we are all used to it.

It is a pattern with repercussions, though: once that becomes normal in a society, it means that people who get to be in power are free to abuse their power much as they want, as long as they are careful enough not to end up in jail. Because, well, nowadays one doesn't need to be decent to stay in power.

I don't need to follow the law to stay in power

That first pattern is already quite well accepted in Italy. So much well accepted, that I think we are starting to see what comes next.

At the end of March we are going to have elections for some regional governors. Funnily enough, in Lazio, the very important region around Rome, the centre-right coalition failed to submit the paperwork on time, and is out of the elections.

It is not just red tape: at some point someone will have to print out the ballots and dispatch them to the voting booths, so one expects to have the coalition logos and the names of the candidates submitted in time, together with signatures supporting the candidates and whatever else the election process needs.

Well, they missed the deadline, they got there after closing time and the building was, well, closed.

It was a fantastic opportunity for a laugh. Memes blossomed on the Italian intarwebs and we now have 2 or 3 new expressions to mean "stupid".

However, now it's hard to tell what is going to happen. On one hand, you can't exclude one of the two major coalitions because of some bureaucratic detail like an office closing time. But on the other hand, several minor coalitions have been excluded in all sorts of past elections because of similar things, and it really would not be fair to start making exceptions now.

But Lombardia, the region around Milan and Emilia Romagna, the one around Bologna, both very, very important, are having similar kinds of problems.

In both regions the previous governors are running again, for the 3rd time in a row, and most likely they legally can't do it, and if elected one can sue and force them to resign, because they have been in power long enough. Lots of paper is being shuffled at the moment to figure if they can get away with it or not.

Oh, and the lawyers of the candidate for the Milan region also managed to get to the tribunal after closing time, but apparently there was still someone inside and they managed to shout loud enough, or somesuch.

Anyway, the situation is getting hot. The Lazio coalition that has been excluded because of their incompetence is now hard at work pushing their potential voters to mount a fracas. Chances are that eventually they'll get away with it, and manage to take part to the election. If that happens, they will likely get close to winning it.

So this seems to be a new pattern that is emerging:

I don't need to follow the law to stay in power

Which, again, is a pattern with quite some repercussions. It is something much more radical than just an issue with morality: it means feudalism, it means we are culturally ready to accept dictatorship.

So, please do me a favour: do not think for a moment that Italy is just a funny place with lemons and tomatoes, and watch out for these patterns emerging around you.

Posted Fri Mar 5 16:52:31 2010 Tags:

Feedback democratico

Le elezioni sono passate, la frustrazione rimane.

Per la comodità di chi volesse dare un democratico feedback alle deludenti forze politiche in gioco, dallo spiegare pacatamente le proprie ragioni per l'aver votato un altro, al mandarli cortesemente a scoreggiare nella farina, ecco un po' di email:

Non dimenticate di mettere bene in chiaro che la mail di insulti che avete mandato non è da intendere come una richiesta di essere iscritti alle loro newsletter.

Posted Tue Jun 9 10:45:05 2009 Tags:

Igiene pubblica

Stamattina:

  • Andare in via Gramsci 12
  • Andare alla porta 10 piano terra
  • Prendere il numero bianco con bordo blu
  • Aspettare (due ore)
  • Fare vaccino
  • Andare alla porta 22 piano terra
  • Pagare
  • Tornare alla porta 10 piano terra
  • Consegnare la ricevuta
  • Ritirare il certificato

Mi sono sentito come la tartaruga del logo.

Posted Sat Jun 6 00:57:39 2009 Tags:

Housing agent in the UK

Yesterday afternoon:

Me: "Where is the gas central heating?"

Agent: "Over there, behind the chimney"

Me: "I see. Beyond that wall is another apartment: where do the fumes go?"

Agent: "Up the chimney"

Me: "Good. And where does it get oxygen from?"

Agent: "It does not need any oxygen, it only needs gas"

Me: "The gas central heater does not have any oxygen or air??"

Agent: "No, it only needs gas"

Me: "I think the laws of physics would have problems with that"

Agent: "You'd have to ask that to British Gas"

Posted Sat Jun 6 00:57:39 2009 Tags:

Italian weather agencies

Most Italian regions have agencies providing good regional weather forecast. However, I could not find a useful national index of all their weather forecast pages. So I made one, tweaking an image map found on wikipedia.

Abruzzo Basilicata Calabria Campania Emilia-Romagna Friuli-Venezia Giulia Lazio Liguria Lombardia Marche Molise Piemonte Puglia Sardegna Sicilia Toscana Trentino-Alto Adige Umbria Valle d'Aosta Veneto

Italia suddivisa per regioni

I only filled in those regions that I was checking to organise a specific trip. With time, if I'll keep using this map I'll add more regions. Of course, if you now of a weather forecast page of a regional, public funded agency that is missing here, send me an e-mail.

Posted Sat Jun 6 00:57:39 2009 Tags:

Etiopia

È interessante, bello e triste allo stesso tempo trovarsi a ridefinire il significato di "Abissinia". E maledire che per i primi 30 anni della tua vita, quella parola l'hai sentita soltanto quando uno stronzo cantava "Faccetta nera".

Posted Sat Jun 6 00:57:39 2009 Tags:
Posted Sat Jun 6 00:57:39 2009 Tags:

My company

I had an interesting conversation in which I was explaining my current work situation. It might be fun to share:

< mornfall> you are working on a timed contract?
< enrico> I'm probably worse than a timed contract: I had to setup a company
< enrico> The government is restricting them so badly that they couldn't hire
          me even temporarily, and they had to buy software from me
< enrico> so I had to become a company
< enrico> I AM THE COMPANY
< mornfall> oh duh
< enrico> I'm my employer
< enrico> and my employee
< enrico> in all this, the government proudly shows statistics of lots of new
          companies being created in Italy as a sign that our economy is so
          good
< mornfall> be nice to your employee :-)
< mornfall> hmh
< enrico> so last friday there was a general strike and my company sent a
          press release to the major labor union telling them that the CEO
          together with all the employees decided to fully take part in the
          strike
< enrico> Signed: Enrico Zini, directory of the company Enrico Zini;
          Employees of the company Enrico Zini: Enrico Zini
< mornfall> that's getting tough
< enrico> I'm very nice to my employees, hes :)
< enrico> but sometimes I wonder if masturbation could be considered sexual
          harassment from my boss
< mornfall> LOL
< mornfall> you need to appoint a secretary
< enrico> I have one: Enrico Zini.  He's a bit messy, though.
< enrico> The only thing which is not called Enrico Zini is the accountant.
< enrico> Incidentally, he's called Vincenzo Zini, but he's isn't a relative
          of mine :)
< mornfall> in that case i think the boss is harassing the secretary not the
          other employee
< enrico> However, he is member of the local LUG and has a Jabber account :)
< mornfall> fun :)
< enrico> oh, ok.  But the employee might be harassing the secretary, or the
          secretary is harassing the employee and the boss
< enrico> IDEA
< enrico> Tonight I'll propose an orgy with the boss, the secretaries and the
          employees!
< enrico> (but not the accountant)

(blogged with mornfall's permission)

Posted Sat Jun 6 00:57:39 2009 Tags:

Altro cazzeggio in letteratura

Dopo il mio talk sul cazzeggio e la versione inglese, il buon godog mi segnala un'altra serie di link di letteratura "futile":

Quoto in toto la facezia LXI:

DI GUGLIELMO CHE AVEVA UN AFFARE ABBONDANTE

Nella città di Terranova eravi un uomo che aveva nome Guglielmo, che facea il falegname ed era assai ben provvisto dalla natura. E la moglie fortunata narrò la cosa alle vicine, e quando questa morì, condusse egli in moglie una giovinetta ingenua, che avea nome Antonia, e che quando fu sposa seppe dai vicini che arma potente possedesse il marito. Nella prima notte che ella fu col marito tremava assai, e voleva sfuggirlo né voleva lasciar fare. E l'uomo capì di che cosa avesse timore la ragazza, e per consolarla le disse che ciò che ella aveva udito dire era vero, ma che egli ne aveva due, uno più grande e uno più piccolo: «E di questo», soggiunse, «per non farti male, mi servirò questa notte; e vedrai che ti farà bene; poi se ti piacerà proveremo col più grande». La ragazza acconsentì e cedette senza pianto e senza dolore all'uomo. E dopo un mese, fattasi più franca e più audace, una notte, mentre accarezzava suo marito: «Amico mio», gli disse, «se ora ti volessi servire di quell'altro ch'è più grande? « E l'uomo, che ne avea quasi quanto un asino, rise dell'appetito della donna; e da lui una volta udii narrare, in compagnia, questa storia.

Posted Sat Jun 6 00:57:39 2009 Tags:

Allucinazioni

Bologna, 14 dicembre 2005.

Stamattina andavo a Bologna in macchina (capita raramente, ma odio quando capita).

Al primo maggio tiro dritto per andare in via Colombo e la coda dell'occhio tira su un cartellone dei saldi da un capannone sulla destra.

C'è qualcosa di strano. Guardo meglio.

"Niente piú telefonate! Svendita totale per cambio gestione"

Sorrido: il capannone era "Il Mobile di Castel Maggiore".

Posted Sat Jun 6 00:57:39 2009 Tags:

Edifici

Da una canzone in amarico:

"Il tuo amore è diventato vecchio

come gli edifici costruiti dagli italiani"

Posted Sat Jun 6 00:57:39 2009 Tags:

Off by one

A few days ago I was around a table having a hot pot and having a small chat with other friends around the table:

X: How old are you?  I was born in 1976
Me: I was born in 1976 as well.  Cool year!

a few minutes pass, I realise something:

Me: So, since you were born in 1976, how old are you?
X: I'm 30
Me: I thought so.  I'm 29 :)

It turns out that in some countries, at the moment you're born you're 1 year old instead of 0: it's just a matter of convention.

And then today it's the first day of 2006. It's also the 95th year since the founding of Republic of China, which is another way of counting years here. The good thing is that if you see that a bottle of milk expires in '95, it means it could still be good.

So, when was Republic of China founded? Of course in 1912, but since when it was founded it was 1 year old, we're now in 95 instead of 94.

Things would be even more interesting if the Lunar Calendar used numbers for the years, but they instead use a "sexagesimal stem-branch cycle" so I cannot say that, according to the lunar calendar, we're still in 2005 until February 29 However, people do have two different birthdays.

And why am I in Taiwan but they count since the founding of Republic of China? I'll tell you if you buy me a glass of wine :)

Anyway, Today is Sweetmorn, the 1st day of Chaos in the YOLD 3172

Posted Sat Jun 6 00:57:39 2009 Tags:
Posted Sat Aug 1 15:48:47 2015

Entries to also be published on truelite.it

Free as in Facebook

Yesterday we were in an airport. We tried to connect to the airport "free" wifi. It had a captive portal that asked for a lot of personal information before one could maybe get on the internet, and we gave up. Bologna Airport, no matter what they do to pretend that they like you, it's always clear that they don't.

I looked at the captive portal screen and I said: «ah yes, "free" wifi. Free as in Facebook».

We figured that we had an expression that will want to be reused.

Posted Mon Mar 9 10:58:49 2015 Tags:

Setting up Akonadi

Now that I have a CalDAV server that syncs with my phone I would like to use it from my desktop.

It looks like akonadi is able to sync with CalDAV servers, so I'm giving it a try.

First thing first is to give a meaning to the arbitrary name of this thing. Wikipedia says it is the oracle goddess of justice in Ghana. That still does not hint at all at personal information servers, but seems quite nice. Ok. I gave up with software having purpose-related names ages ago.

# apt-get install akonadi-server akonadi-backend-postgresql

Akonadi wants a SQL database as a backend. By default it uses MySQL, but I had enough of MySQL ages ago.

I tried SQLite but the performance with it is terrible. Terrible as in, it takes 2 minutes between adding a calendar entry and having it show up in the calendar. I'm fascinated by how Akonadi manages to use SQLite so badly, but since I currently just want to get a job done, next in line is PostgreSQL:

# su - postgres
$ createuser enrico
$ psql postgres
postgres=# alter user enrico createdb;

Then as enrico:

$ createdb akonadi-enrico
$ cat <<EOT > ~/.config/akonadi/akonadiserverrc
[%General]
Driver=QPSQL

[QPSQL]
Name=akonadi-enrico
StartServer=false
Host=
Options=
ServerPath=
InitDbPath=

I can now use kontact to connect Akonadi to my CalDAV server and it works nicely, both with calendar and with addressbook entries.

KDE has at least two clients for Akonadi: Kontact, which is a kitchen sink application similar to Evolution, and KOrganizer, which is just the calendar and scheduling component of Kontact.

Both work decently, and KOrganizer has a pretty decent startup time. I now have a usable desktop PIM application that is synced with my phone. W00T!

Next step is to port my swift little calendar display tool to use Akonadi as a back-end.

Posted Tue Feb 17 15:34:55 2015 Tags:

seat-inspect

Four months ago I wrote this somewhere:

Seeing a DD saying "this new dbus stuff scares me" would make most debian users scared. Seeing a DD who has an idea of what is going on, and who can explain it, would be an interesting and exciting experience.

So, let's be exemplary, competent and patient. Or at least, competent. Some may like or not like the changes, but do we all understand what is going on? Will we all be able to support our friends and customers running jessie?

I confess that although I understand the need for it, I don't feel competent enough to support systemd-based machines right now.

So, are we maybe in need of help, cheat sheets, arsenals of one-liners, diagnostic tools?

Maybe a round of posts on -planet like "one debian package a day" but with new features that jessie will have, and how to understand them and take advantage of them?

That was four months ago. In the meantime, I did some work, and it got better for me.

Yesterday, however, I've seen an experienced Linux person frustrated because the shutdown function of the desktop was doing nothing whatsoever. Today I found John Goerzen's post on planet.

I felt like some more diagnostic tools were needed, so I spent the day making seat-inspect.

seat-inspect tries to make the status of the login/seat system visible, to help with understanding and troubleshooting.

The intent of running the code is to have an overview of the system status, both to see what the new facilities are about, and to figure out if there is something out of place.

The intent of reading the code is to have an idea of how to use these facilities: the code has been written to be straightforward and is annotated with relevant bits from the logind API documentation.

seat-inspect is not a finished tool, but a starting point. I put it on github hoping that people will fork it and add their own extra sanity checks and warnings, so that it can grow into a standard thing to run if a system acts weird.

As it is now, it should be able to issue warnings if some bits are missing for network-manager or shutdown functions to work correctly. I haven't really tested that, though, because I don't have a system at hand where they are currently not working fine.

Another nice thing of it is that when running seat-inspect -v you get a dump of what logind/consolekit think about your system. I found it an interesting way to explore the new functionalities that we recently grew. The same can be done, and in more details, with loginctl calls, but I lacked a summary.

After writing this I feel a bit more competent, probably enough to sit at somebody's computer and poke into loginctl bits. I highly recommend the experience.

Posted Tue Feb 10 18:06:43 2015 Tags:

Playing with python, terminfo and command output

I am experimenting with showing progress on the terminal for a subcommand that is being run, showing what is happening without scrolling away the output of the main program, and I came out with this little toy. It shows the last X lines of a subcommand output, then gets rid of everything after the command has ended.

Usability-wise, it feels like a tease to me: it looks like I'm being shown all sorts of information then they are taken away from me before I managed to make sense of them. However, I find it cute enough to share:

#!/usr/bin/env python3
#coding: utf-8
# Copyright 2015 Enrico Zini <enrico@enricozini.org>.  Licensed under the terms
# of the GNU General Public License, version 2 or any later version.

import argparse
import fcntl
import select
import curses
import contextlib
import subprocess
import os
import sys
import collections
import shlex
import shutil
import logging

def stream_output(proc):
    """
    Take a subprocess.Popen object and generate its output, line by line,
    annotated with "stdout" or "stderr". At process termination it generates
    one last element: ("result", return_code) with the return code of the
    process.
    """
    fds = [proc.stdout, proc.stderr]
    bufs = [b"", b""]
    types = ["stdout", "stderr"]
    # Set both pipes as non-blocking
    for fd in fds:
        fcntl.fcntl(fd, fcntl.F_SETFL, os.O_NONBLOCK)
    # Multiplex stdout and stderr with different prefixes
    while len(fds) > 0:
        s = select.select(fds, (), ())
        for fd in s[0]:
            idx = fds.index(fd)
            buf = fd.read()
            if len(buf) == 0:
                fds.pop(idx)
                if len(bufs[idx]) != 0:
                    yield types[idx], bufs.pop(idx)
                types.pop(idx)
            else:
                bufs[idx] += buf
                lines = bufs[idx].split(b"\n")
                bufs[idx] = lines.pop()
                for l in lines:
                    yield types[idx], l
    res = proc.wait()
    yield "result", res

@contextlib.contextmanager
def miniscreen(has_fancyterm, name, maxlines=3, silent=False):
    """
    Show the output of a process scrolling in a portion of the screen.

    has_fancyterm: true if the terminal supports fancy features; if false, just
    write lines to standard output

    name: name of the process being run, to use as a header

    maxlines: maximum height of the miniscreen

    silent: do nothing whatsoever, used to disable this without needing to
            change the code structure

    Usage:
        with miniscreen(True, "my process", 5) as print_line:
            for i in range(10):
                print_line(("stdout", "stderr")[i % 2], "Line #{}".format(i))
    """
    if not silent and has_fancyterm:
        # Discover all the terminal control sequences that we need
        output_normal = str(curses.tigetstr("sgr0"), "ascii")
        output_up = str(curses.tigetstr("cuu1"), "ascii")
        output_clreol = str(curses.tigetstr("el"), "ascii")
        cols, lines = shutil.get_terminal_size()
        output_width = cols

        fg_color = (curses.tigetstr("setaf") or
                    curses.tigetstr("setf") or "")
        sys.stdout.write(str(curses.tparm(fg_color, 6), "ascii"))

        output_lines = collections.deque(maxlen=maxlines)

        def print_lines():
            """
            Print the lines in our buffer, then move back to the beginning
            """
            sys.stdout.write("{} progress:".format(name))
            sys.stdout.write(output_clreol)
            for msg in output_lines:
                sys.stdout.write("\n")
                sys.stdout.write(msg)
                sys.stdout.write(output_clreol)
            sys.stdout.write(output_up * len(output_lines))
            sys.stdout.write("\r")

        try:
            print_lines()

            def _progress_line(type, line):
                """
                Print a new line to the miniscreen
                """
                # Add the new line to our output buffer
                msg = "{} {}".format("." if type == "stdout" else "!", line)
                if len(msg) > output_width - 4:
                    msg = msg[:output_width - 4] + "..."
                output_lines.append(msg)
                # Update the miniscreen
                print_lines()

            yield _progress_line

            # Clear the miniscreen by filling our ring buffer with empty lines
            # then printing them out
            for i in range(maxlines):
                output_lines.append("")
            print_lines()
        finally:
            sys.stdout.write(output_normal)
    elif not silent:
        def _progress_line(type, line):
            print("{}: {}".format(type, line))
        yield _progress_line
    else:
        def _progress_line(type, line):
            pass
        yield _progress_line

def run_command_fancy(name, cmd, env=None, logfd=None, fancy=True, debug=False):
    quoted_cmd = " ".join(shlex.quote(x) for x in cmd)
    log.info("%s running command %s", name, quoted_cmd)
    if logfd: print("runcmd:", quoted_cmd, file=logfd)

    # Run the script itself on an empty environment, so that what was
    # documented is exactly what was run
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)

    with miniscreen(fancy, name, silent=debug) as progress:
        stderr = []
        for type, val in stream_output(proc):
            if type == "stdout":
                val = val.decode("utf-8")
                if logfd: print("stdout:", val, file=logfd)
                log.debug("%s stdout: %s", name, val)
                progress(type, val)
            elif type == "stderr":
                val = val.decode("utf-8")
                if logfd: print("stderr:", val, file=logfd)
                stderr.append(val)
                log.debug("%s stderr: %s", name, val)
                progress(type, val)
            elif type == "result":
                if logfd: print("retval:", val, file=logfd)
                log.debug("%s retval: %d", name, val)
                retval = val

    if retval != 0:
        lastlines = min(len(stderr), 5)
        log.error("%s exited with code %s", name, retval)
        log.error("Last %d lines of standard error:", lastlines)
        for line in stderr[-lastlines:]:
            log.error("%s: %s", name, line)

    return retval


parser = argparse.ArgumentParser(description="run a command showing only a portion of its output")
parser.add_argument("--logfile", action="store", help="specify a file where the full execution log will be written")
parser.add_argument("--debug", action="store_true", help="debugging output on the terminal")
parser.add_argument("--verbose", action="store_true", help="verbose output on the terminal")
parser.add_argument("command", nargs="*", help="command to run")
args = parser.parse_args()

if args.debug:
    loglevel = logging.DEBUG
elif args.verbose:
    loglevel = logging.INFO
else:
    loglevel = logging.WARN
logging.basicConfig(level=loglevel, stream=sys.stderr)
log = logging.getLogger()

fancy = False
if not args.debug and sys.stdout.isatty():
    curses.setupterm()
    if curses.tigetnum("colors") > 0:
        fancy = True

if args.logfile:
    logfd = open("output.log", "wt")
else:
    logfd = None

retval = run_command_fancy("miniscreen example", args.command, logfd=logfd)

sys.exit(retval)
Posted Wed Jan 21 11:13:31 2015 Tags:

Upgrade Cyanogenmod with an encrypted phone

Cyanogenmod found an update, it downloaded it, then it rebooted to install it and nothing happened. It turns out that the update procedure cannot work if the zip file to install is in encrypted media, so a workaround is to move the zip into unencrypted external storage.

As far as I know, my Nexus 4 has no unencrypted external storage.

This is how I managed to upgrade it, I write it here so I can find it next time:

  1. enable USB debugging
  2. adb pull /cmupdater/cm-11-20141115-SNAPSHOT-M12-mako.zip
  3. adb reboot recovery
  4. choose "install zip from sideload"
  5. adb sideload cm-11-20141115-SNAPSHOT-M12-mako.zip
Posted Fri Dec 19 10:21:29 2014 Tags:

Radicale and DAVDroid

radicale and DAVdroid appeal to me. Let's try to make the whole thing work.

A self-signed SSL certificate

Generating the certificate:

    openssl req -nodes -x509 -newkey rsa:2048 -keyout cal-key.pem -out cal-cert.pem -days 3650
    [...]
    Country Name (2 letter code) [AU]:IT
    State or Province Name (full name) [Some-State]:Bologna
    Locality Name (eg, city) []:
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:enricozini.org
    Organizational Unit Name (eg, section) []:
    Common Name (e.g. server FQDN or YOUR name) []:cal.enricozini.org
    Email Address []:postmaster@enricozini.org

Installing it on my phone:

    openssl x509 -in cal-cert.pem -outform DER -out cal-cert.crt
    adb push cal-cert.crt /mnt/sdcard/
    enrico --follow-instructions http://davdroid.bitfire.at/faq/entry/importing-a-certificate

Installing radicale in my VPS

An updated radicale package, with this patch to make it work with DAVDroid:

    apt-get source radicale
    # I reviewed 063f7de7a2c7c50de5fe3f8382358f9a1124fbb6
    git clone https://github.com/Kozea/Radicale.git
    Move the python code from git to the Debian source
    dch -v 0.10~enrico  "Pulled in the not yet released 0.10 work from upstream"
    debuild -us -uc -rfakeroot

Install the package:

    # dpkg -i python-radicale_0.10~enrico0-1_all.deb
    # dpkg -i radicale_0.10~enrico0-1_all.deb

Create a system user to run it:

    # adduser --system --disabled-password radicale

Configure it for mod_wsgi with auth done by Apache:

    # For brevity, this is my config file with comments removed

    [storage]
    # Storage backend
    # Value: filesystem | multifilesystem | database | custom
    type = filesystem

    # Folder for storing local collections, created if not present
    filesystem_folder = /var/lib/radicale/collections

    [logging]
    config = /etc/radicale/logging

Create the wsgi file to run it:

    # mkdir /srv/radicale
    # cat <<EOT > /srv/radicale/radicale.wsgi
    import radicale
    radicale.log.start()
    application = radicale.Application()
    EOT
    # chown radicale.radicale /srv/radicale/radicale.wsgi
    # chmod 0755 /srv/radicale/radicale.wsgi

Make radicale commit to git

    # apt-get install python-dulwich
    # cd /var/lib/radicale/collections
    # git init
    # chown radicale.radicale -R /var/lib/radicale/collections/.git

Apache configuration

Add a new site to apache:

    $ cat /etc/apache2/sites-available/cal.conf
    # For brevity, this is my config file with comments removed
    <IfModule mod_ssl.c>
    <VirtualHost *:443>
            ServerName cal.enricozini.org
            ServerAdmin enrico@enricozini.org

            Alias /robots.txt /srv/radicale/robots.txt
            Alias /favicon.ico /srv/radicale/favicon.ico

            WSGIDaemonProcess radicale user=radicale group=radicale threads=1 umask=0027 display-name=%{GROUP}
            WSGIProcessGroup radicale
            WSGIScriptAlias / /srv/radicale/radicale.wsgi

            <Directory /srv/radicale>
                    # WSGIProcessGroup radicale
                    # WSGIApplicationGroup radicale
                    # WSGIPassAuthorization On
                    AllowOverride None
                    Require all granted
            </Directory>

            <Location />
                    AuthType basic
                    AuthName "Enrico's Calendar"
                    AuthBasicProvider file
                    AuthUserFile /usr/local/etc/radicale/htpasswd
                    Require user enrico
            </Location>

            ErrorLog{APACHE_LOG_DIR}/cal-enricozini-org-error.log
            LogLevel warn

            CustomLog{APACHE_LOG_DIR}/cal-enricozini-org-access.log combined

            SSLEngine on
            SSLCertificateFile    /etc/ssl/certs/cal.pem
            SSLCertificateKeyFile /etc/ssl/private/cal.key
    </VirtualHost>
    </IfModule>

Then enable it:

    # a2ensite cal.conf
    # service apache2 reload

Create collections

DAVdroid seems to want to see existing collections on the server, so we create them:

    $ apt-get install cadaver
    $ cat <<EOT > /tmp/empty.ics
    BEGIN:VCALENDAR
    VERSION:2.0
    END:VCALENDAR
    EOT
    $ cat <<EOT > /tmp/empty.vcf
    BEGIN:VCARD
    VERSION:2.1
    END:VCARD
    EOT
    $ cadaver https://cal.enricozini.org
    WARNING: Untrusted server certificate presented for `cal.enricozini.org':
    [...]
    Do you wish to accept the certificate? (y/n) y
    Authentication required for Enrico's Calendar on server `cal.enricozini.org':
    Username: enrico
    Password: ****
    dav:/> cd enrico/contacts.vcf/
    dav:/> put /tmp/empty.vcf
    dav:/> cd ../calendar.ics/
    dav:/> put /tmp/empty.ics
    dav:/enrico/calendar.ics/> ^D
    Connection to `cal.enricozini.org' closed.

DAVdroid configuration

  1. Add a new DAVdroid sync account
  2. Use server/username configuration
  3. For server, use https:////
  4. Add username and password

It should work.

Related links

Posted Tue Dec 9 16:35:50 2014 Tags:

Alternate rescue boot entry with systemd

Since systemd version 215, adding systemd.debug-shell to the kernel command line activates the debug shell on tty9 alongside the normal boot. I like the idea of that, and I'd like to have it in my standard 'rescue' entry in my grub menu.

Unfortunately, by default update-grub does not allow to customize the rescue menu entry options. I have just filed #766530 hoping for that to change.

After testing the patch I proposed for /etc/grub.d/10_linux, I now have this in my /etc/default/grub, with some satisfaction:

GRUB_CMDLINE_LINUX_RECOVERY="systemd.log_target=kmsg systemd.log_level=debug systemd.debug-shell"

Further information:

Thanks to sjoerd and uau on #debian-systemd for their help.

Posted Thu Oct 23 22:06:30 2014 Tags:

Spelling a chilometri zero

Lo spelling internazionale è troppo globalizzato, e volete recuperare un attimo la dimensione del posto dove siete nati e cresciuti?

Da oggi c'è questo script che fa per voi: gli dite dove abitate, e lui vi crea lo spelling a chilometri zero.

$ git clone git@gitorious.org:trespolo/osmspell.git
$ cd osmspell
$ ./osmspell "San Giorgio di Piano"
1: San Giorgio di Piano, BO, EMR, Italia
2: San Giorgio di Piano, Via Codronchi, San Giorgio di Piano, BO, EMR, Italia
3: San Giorgio Di Piano, Via Libertà, San Giorgio di Piano, BO, EMR, Italia
Choose one: 1
Center: 44.6465332, 11.3790398
A Argelato, Altedo
B Bentivoglio, Bologna, Boschi
C Cinquanta, Castagnolo Minore, Castel Maggiore, Cento
D Dosso
E Eremo di Tizzano
F Funo di Argelato, Finale Emilia, Ferrara, Fiesso
G Gherghenzano, Galliera, Gesso
I Il Cucco, Irnerio, Idice
L Località Fortuna, Lovoleto, Lippo
M Malacappa, Massumatico, Minerbio, Marano
N Navile
O Osteriola, Ozzano dell'Emilia, Oca
P Piombino, Padulle, Poggio Renatico, Piave
Q Quarto Inferiore, Quattrina
R Rubizzano, Renazzo, Riale
S San Giorgio di Piano, Saletto
T Torre Verde, Tintoria, Tombe
U Uccellino
V Venezzano Mascarino, Vigarano Mainarda, Veduro
X XII Morelli
Z Zenerigolo, Zola Predosa

I dati vengono da OSM, e lo script è un ottimo esempio di come usarne la API di geolocazione (veloci) e la API di query geografica (lenta).

Posted Sat Jan 4 00:38:16 2014 Tags:
Posted Tue Apr 7 19:20:12 2015

Rants, kept to a bare minimum and strictly with a useful component.

Billing an Italian public administration

Here's a simple guide for how I managed to bill one of my customers as is now mandated by law in Italy.

Create a new virtualbox machine

I would never do any of this to any system I would ever want to use for anything else, so it's virtual machine time.

  • I started virtualbox, created a new machine for Ubuntu 32bit, 8Gb disk, 4Gb RAM, and placed the .vdi image in an encrypted partition. The web services of Infocert's fattura-pa requires "Java (JRE) a 32bit di versione 1.6 o superiore".
  • I installed Ubuntu 12.04 on it: that is what dike declares to support.
  • I booted the VM, installed virtualbox-guest-utils, and de sure I also had virtualbox-guest-x11
  • I restarted the VM so that I could resize the virtualbox window and have Ubuntu resize itself as well. Now I could actually read popup error messages in full.
  • I changed the desktop background to something that gave me the idea that this is an untrusted machine where I need to be very careful of what I type. I went for bright red.

Install smart card software into it

  • apt-get install pcscd pcsc-tools opensc
  • In virtualbox, I went to Devices/USB devices and enabled the smart card reader in the virtual machine.
  • I ran pcsc_scan to see if it could see my smart card.
  • I ran Firefox, went to preferences, advanced, security devices, load. Module name is "CRS PKCS#11", module path is /usr/lib/opensc-pkcs11.so
  • I went to https://fattura-pa.infocamere.it/fpmi/service and I was able to log in. To log in, I had to type the PIN 4 times into popups that offered little explanations about what was going on, enjoying cold shivers because the smart card would lock itself at the 3rd failed attempt.
  • Congratulations to myself! I thought that all was set, but unfortunately, at this stage, I was not able to do anything else except log into the website.

Descent into darkness

Set up things for fattura-pa

  • I got the PDF with the setup instructions from here. Get it too, for a reference, a laugh, and in case you do not believe the instructions below.
  • I went to https://www.firma.infocert.it/installazione/certificato.php, and saved the two certificates.
  • Firefox, preferences, advanced, show certificates, I imported both CA certificates, trusted for everything, all my base are belong to them.
  • apt-get install icedtea-plugin
  • I went to https://fattura-pa.infocamere.it/fpmi/service and tried to sign. I could not: I got an error about invalid UTF8 for something or other in Firefox's stdandard error. Firefox froze and had to be killed.

Set up things for signing locally with dike

  • I removed icedtea so that I could use the site without firefox crashing.
  • I installed DiKe For Ubuntu 12.04 32bit
  • I ran dikeutil to see if it could talk to my smart card
  • When signing with the website, I chose the manual signing options and downloaded the zip file with the xml to be signed.
  • I got a zip file, unzipped it.
  • I loaded the xml into dike.
  • I signed it with dike.
  • I got this error message: "nessun certificato di firma presente sul dispositivo di firma" and then this error message: "Impossibile recuperare il certificato dal dispositivo di firma". No luck.

Set up things for signing locally with ArubaSign

  • I went to https://www.pec.it/Download.aspx
  • I downloaded ArubaSign for Linux 32 bit.
  • Oh! People say that it only works with Oracle's version of Java.
  • sudo add-apt-repository ppa:webupd8team/java
  • apt-get update
  • apt-get install oracle-java7-installer
  • During the installation process I had to agree to also sell my soul to Oracle.
  • tar axf ArubaSign*.tar*
  • cd ArubaSing-*/apps/dist
  • java -jar ArubaSign.jar
  • I let it download its own updates. Another time I did not. It does not seem to matter: I get asked that question every time I start it anyway.
  • I enjoyed the fancy brushed metal theme, and had an interesting time navigating an interface where every label on every icon or input field was truncated.
  • I downloaded https://www.pec.it/documenti/Manuale_ArubaSign2_firma%20Remota_V03_02_07_2012.pdf to get screenshots of that interface with all the labels intact
  • I signed the xml that I got from the website. I got told that I needed to really view carefully what I was signing, because the signature would be legally binding
  • I enjoyed carefully reading a legally binding, raw XML file.
  • I told it to go ahead, and there was now a .p7m file ready for me. I rejoiced, as now I might, just might actually get paid for my work.

Try fattura-pa again

Maybe fattura-pa would work with Oracle's Java plugin?

  • I went to https://fattura-pa.infocamere.it/fpmi/service
  • I got asked to verify java at www.java.com. I did it.
  • I told FireFox to enable java.
  • Suddenly, and while I was still in java.com's tab, I got prompted about allowing Infocert's applet to run: I allowed it to run.
  • I also got prompted several times, still while the current tab was not even Infocert's tab, about running components that could compromise the security of my system. I allowed and unblocked all of them.
  • I entered my PIN.
  • Congratulations! Now I have two ways of generating legally binding signatures with government issued smart cards!

Aftermath

I shut down that virtual machine and I'm making sure I never run anything important on it. Except, of course, generating legally binding signatures as required by the Italian government.

What could possibly go wrong?

Posted Thu Jul 2 23:48:36 2015 Tags:

debtags rewritten in python3

In my long quest towards closing #540218, I have uploaded a new libept to experimental. Then I tried to build debtags on a sid+experimental chroot and the result runs but has libc's free() print existential warnings about whatevers.

At a quick glance, there are now things around like a new libapt, gcc 5 with ABI changes, and who knows what else. I figured how much time it'd take me to debug something like that, and I've used that time to rewrite debtags in python3. It took 8 hours, 5 of pleasant programming and the usual tax of another 3 of utter frustration packaging the results. I guess I gained over the risk of spending an unspecified amount of hours of just pure frustration.

So from now on debtags is going to be a pure python3 package, with dependencies on only python3-apt and python3-debian. 700 lines of python instead of several C++ files built on 4 layers of libraries. Hopefully, this is the last of the big headaches I get from hacking on this package. Also, one less package using libept.

Posted Sun Jun 21 18:04:39 2015 Tags:

Another day in the life of a poor developer

try:
    # After Python 3.3
    from collections.abc import Iterable
except ImportError:
    # This has changed in Python 3.3 (why, oh why?), reinforcing the idea that
    # the best Python version ever is still 2.7, simply because upstream has
    # promised that they won't touch it (and break it) for at least 5 more
    # years.
    from collections import Iterable

import shlex
if hasattr(shlex, "quote"):
    # New in version 3.3.
    shell_quote = shlex.quote
else:
    # Available since python 1.6 but deprecated since version 2.7: Prior to Python
    # 2.7, this function was not publicly documented. It is finally exposed
    # publicly in Python 3.3 as the quote function in the shlex module.
    #
    # Except everyone was using it, because it was the only way provided by the
    # python standard library to make a string safe for shell use
    #
    # See http://stackoverflow.com/questions/35817/how-to-escape-os-system-calls-in-python
    import pipes
    shell_quote = pipes.quote

import shutil
if hasattr(shutil, "which"):
    # New in version 3.3.
    shell_which = shutil.which
else:
    # Available since python 1.6:
    # http://stackoverflow.com/questions/377017/test-if-executable-exists-in-python
    from distutils.spawn import find_executable
    shell_which = find_executable
Posted Fri Feb 27 12:02:33 2015 Tags:

Akonadi client example

After many failed attemps I have managed to build a C++ akonadi client. It has felt like one of the most frustrating programming experiences of my whole life, so I'm sharing the results hoping to spare others from all the suffering.

First thing first, akonadi client libraries are not in libakonadi-dev but in kdepimlibs5-dev, even if kdepimlibs5-dev does not show in apt-cache search akonadi.

Then, kdepimlibs is built with Qt4. If your application uses Qt5 (mine was) you need to port it back to Qt4 if you want to talk to Akonadi.

Then, kdepimlibs does not seem to support qmake and does not ship pkg-config .pc files, and if you want to use kdepimlibs your build system needs to be cmake. I ported by code from qmake to cmake, and now qtcreator wants me to run cmake by hand every time I change the CMakeLists.txt file, and it stopped allowing to add, rename or delete sources.

Finally, most of the code / build system snippets found on the internet seem flawed in a way or another, because the build toolchain of Qt/KDE applications has undergone several redesignins during time, and the network is littered with examples from different eras. The way to obtain template code to start a Qt/KDE project is to use kapptemplate. I have found no getting started tutorial on the internet that said "do not just copy the snippets from here, run kapptemplate instead so you get them up to date".

kapptemplate supports building an "Akonadi Resource" and an "Akonadi Serializer", but it does not support generating template code for an akonadi client. That left me with the feeling that I was dealing with some software that wants to be developed but does not want to be used.

Anyway, now an example of how to interrogate Akonadi exists as is on the internet. I hope that all the tears of blood that I cried this morning have not been cried in vain.

Posted Mon Feb 23 15:44:01 2015 Tags:

The wonders of missing documentation

Update: I have managed to build an example Akonadi client application.

I'm new here, I want to make a simple C++ GUI app that pops up a QCalendarWidget which my local Akonadi has appointments.

I open qtcreator, create a new app, hack away for a while, then of course I get undefined references for all Akonadi symbols, since I didn't tell the build system that I'm building with akonadi. Ok.

How do I tell the build system that I'm building with akonadi? After 20 minutes of frantic looking around the internet, I still have no idea.

There is a package called libakonadi-dev which does not seem to have anything to do with this. That page mentions everything about making applications with Akonadi except how to build them.

There is a package called kdepimlibs5-dev which looks promising: it has no .a files but it does have haders and cmake files. However, qtcreator is only integrated with qmake, and I would really like the handholding of an IDE at this stage.

I put something together naively doing just what looked right, and I managed to get an application that segfaults before main() is even called:

/*
 * Copyright © 2015 Enrico Zini <enrico@enricozini.org>
 *
 * This work is free. You can redistribute it and/or modify it under the
 * terms of the Do What The Fuck You Want To Public License, Version 2,
 * as published by Sam Hocevar. See the COPYING file for more details.
 */
#include <QDebug>

int main(int argc, char *argv[])
{
    qDebug() << "BEGIN";
    return 0;
}
QT       += core gui widgets
CONFIG += c++11

TARGET = wtf
TEMPLATE = app

LIBS += -lkdecore -lakonadi-kde

SOURCES += wtf.cpp

I didn't achieve what I wanted, but I feel like I achieved something magical and beautiful after all.

I shall now perform some haruspicy on those oscure cmake files to see if I can figure something out. But seriously, people?

Posted Mon Feb 23 11:36:18 2015 Tags:

Mozilla marketplace facepalm

This made me sad.

My view, which didn't seem to be considered in that discussion, is that people concerned about software freedom and security are likely to stay the hell away from such an app market and its feedback forms.

Also, that thread made me so sad about the state of that developer community that I seriously do not feel like investing energy into going through the hoops of getting an account in their bugtracker to point this out.

Sigh.

Posted Fri Jan 23 15:13:16 2015 Tags:

SSL "protection"

In my experience with my VPS, setting up pretty much any service exposed to the internet, even a simple thing to put a calendar in my phone requires an SSL certificate, which costs money, which needs to be given to some corporation or another.

When the only way to get protection from a threat is to give money to some big fish, I feel like I'm being forced to pay protection money.

I look forward to this.

Posted Thu Dec 11 15:35:25 2014 Tags:

Laptop, I demand that you suspend!

Dear Lazyweb,

Sometimes some application prevents suspend on my laptop. I want to disable that feature: how?

I understand that there may exist some people who like that feature. I, on the other hand, consider a scenario like this inconceivable:

  1. I'm on a plane working with my laptop, the captain announces preparations for landing, so I quickly hit the suspend button (or close the lid) on my laptop and stow it away.
  2. One connecting flight later, I pick up my backpack, I feel it unusually hot and realise that my laptop has been on all along, and is now dead from either running out of battery or thermal protection.
  3. I think things that, if spoken aloud in front of a pentacle, might invoke major lovecraftian horrors.

I do not want this scenario to ever be possible. I want my suspend button to suspend the laptop no matter what. If a process does not agree, I'm fine with suspending it anyway, or killing it.

If I want my laptop to suspend, I generally have a good enough real-world reason for it, and I cannot conceive that a software could ever be allowed to override my command.

How do I change this? I don't know if I should look into systemd, upowerd, pm-utils, the kernel, the display manager or something else entirely. I worry that I cannot even figure where to start looking for a solution.

This happened to me multiple times already, and I consider it ridiculous. I know that it can cause me data loss. I know that it can cause me serious trouble in case I was relying on having some battery or state left at my arrival. I know that depending on what is in my backpack, this could also be physically dangerous.

So, what knob do I tweak for this? How do I make suspend reliable?

Update

Systemd has an inhibitor system, and systemd-inhibit --list only lists 'delay' blocks in my system. It is an interesting feature that seems to be implemented in the right way, and it could mean that I finally can get my screen to be locked before the system is suspended.

It is possible to configure the inhibitor system in /etc/systemd/logind.conf, including ways to ignore inhibitors, and a maximum time after which inhibitors are ignored if not yet released.

Try as I might to run everything that I was running on the plane that time, I could not manage to see anything take an inhibitor block that could have prevented my suspend. I now suspect that what happened to me was a glitch caused by something else (hardware? kernel? cosmic rays!) during that specific suspend.

When I had this issue in the past it looks like the infrastructure at the time was far more primitive that what we have now with systemd, so I guess that when writing my blog post I had simply correlated my old experiences with a one-off suspend glitch.

If I want to investigate or tune further, to test the situation with a runaway block, I can use commands like systemd-inhibit --mode=block sleep 3600.

I'm quite happy to see that we're moving to a standard and sane system for this. In the meantime, I have learnt that pm-utils has now become superfluous and can be deinstalled, and so can acpi-support and acpi-support-base.

Thanks vbernat, mbiebl, and ah, on #debian-devel for all the help.

Posted Thu Sep 11 14:32:40 2014 Tags:

Wheezy for industrial software development

I'm helping with setting up a wheezy-based toolchain for industrial automation.

The basic requirements are: live-build, C++11, Qt 5.3, and a frozen internal wheezy mirror.

debmirror

A good part of a day's work was lost because of #749734 and possibly #628779. Mirror rebuild is still ongoing, and fingers crossed.

This is Italy, and you can't simply download 21Gb of debs just to see how it goes.

C++11

Stable toolchains for C++11 now exist and have gained fast adoption. It makes sense, since given what is in C++11 it is unthinkable to start a new C++ project with the old standard nowadays.

C++11 is supported by g++ 4.8+ or clang 3.3+. None of them is available on wheezy or wheezy-backports.

Backports exist of g++ 4.8 only for Ubuntu 12.04, but they are uninstallable on wheezy due at least to a different libc6. I tried rebuilding g++4.8 on wheezy but quickly gave up.

clang 3.3 has a build dependency on g++ 4.8. LOL.

However, LLVM provides an APT repository with their most recent compiler, and it works, too. C++11 problem solved!

Qt 5.3

Qt 5.3 is needed because of the range of platforms it can target. There is no wheezy backport that I can find.

I cannot simply get it from Qt's Download page and install it, since we need it packaged, to build live ISOs with it.

I'm attempting to backport the packages from experimental to wheezy.

Here are its build dependencies:

libxcb-1.10 (needed by qt5)

Building this is reasonably straightforward.

libxkbcommon 0.4.0 (needed by qt5)

The version from jessie builds fine on wheezy, provided you remove --fail-missing from the dh_install invocation.

libicu 52.1 (needed by harfbuzz)

The jessie packages build on wheezy, provided that mentions of clang are deleted from source/configure.ac, since it fails to build with clang 3.5 (the one currently available for wheezy on llvm.org).

libharfbuzz-dev

Backporting this is a bloodbath: the Debian packages from jessie depend on a forest of gobject hipsterisms of doom, all unavailable on wheezy. I gave up.

qt 5.3

qtbase-opensource-src-5.3.0+dfsg can be made to build with an embedded version of harfbuzz, with just this change:

diff -Naur a/debian/control a/debian/control
--- a/debian/control    2014-05-20 18:48:27.000000000 +0200
+++ b/debian/control    2014-05-29 17:45:31.037215786 +0200
@@ -28,7 +28,6 @@
                libgstreamer-plugins-base0.10-dev,
                libgstreamer0.10-dev,
                libgtk2.0-dev,
-               libharfbuzz-dev,
                libicu-dev,
                libjpeg-dev,
                libmysqlclient-dev,
diff -Naur a/debian/rules b/debian/rules
--- a/debian/rules  2014-05-18 01:56:37.000000000 +0200
+++ b/debian/rules  2014-05-29 17:45:25.738634371 +0200
@@ -108,7 +108,6 @@
                -plugin-sql-tds \
                -system-sqlite \
                -platform $(platform_arg) \
-               -system-harfbuzz \
                -system-zlib \
                -system-libpng \
                -system-libjpeg \

(thanks Lisandro Damián Nicanor Pérez Meyer for helping me there!)

There are probably going to be further steps in the Qt5 toolchain.

Actually, let's try prebuilt binaries

The next day with a fresh mind we realised that it is preferable to reduce our tampering with the original wheezy to a minimum. Our current plan is to use wheezy's original Qt and Qt-using packages, and use Qt's prebuilt binaries in /opt for all our custom software.

We run Qt's installer, tarred the result, and wrapped it in a Debian package like this:

$ cat debian/rules
#!/usr/bin/make -f

QT_VERSION = 5.3

%:
    dh $@

override_dh_auto_build:
    dh_auto_build
    sed -re 's/@QT_VERSION@/$(QT_VERSION)/g' debian-rules.inc.in > debian-rules.inc

override_dh_auto_install:
    dh_auto_install
    # Download and untar the prebuild Qt5 binaries
    install -d -o root -g root -m 0755 debian/our-qt5-sdk/opt/Qt
    curl http://localserver/Qt$(QT_VERSION).tar.xz | xz -d | tar -C debian/our-qt5-sdk/opt -xf -
    # Move the runtime part to our-qt5
    install -d -o root -g root -m 0755 debian/our-qt5/opt/Qt
    mv debian/our-qt5-sdk/opt/Qt/$(QT_VERSION) debian/our-qt5/opt/Qt/
    # Makes dpkg-shlibdeps work on packages built with Qt from /opt
    # Hack. Don't try this at home. Don't ever do this unless you
    # know what you are doing. This voids your warranty. If you
    # know what you are doing, you won't do this.
    find debian/our-qt5/opt/Qt/$(QT_VERSION)/gcc_64/lib -maxdepth 1 -type f -name "lib*.so*" \
        | sed -re 's,^.+/(lib[^.]+)\.so.+$$,\1 5 our-qt5 (>= $(QT_VERSION)),' > debian/our-qt5.shlibs


$ cat debian-rules.inc.in
export PATH := /opt/Qt/@QT_VERSION@/gcc_64/bin:$(PATH)
export QMAKESPEC=/opt/Qt/@QT_VERSION@/gcc_64/mkspecs/linux-clang/

To build one of our packages using Qt5.3 and clang, we just add this to its debian/rules:

include /usr/share/our-qt5/debian-rules.inc

Wrap up

We got the dependencies sorted. Hopefully the mirror will rebuild itself tonight and tomorrow we can resume working on our custom live system.

Posted Thu May 29 18:05:17 2014 Tags:

Shops

Christmas songs should only ever be played on Christmas day.

In church.

At midnight.

Unless I happen to be there.

Posted Mon Dec 2 14:07:58 2013 Tags:
Posted Sat Jun 6 00:57:39 2009

Pages about Debtags.

Evolution's old odd mail folders to mbox

Something wrong happened in my dad's Evolution. It just would get stuck checking mail forever, with no useful diagnostic that I could find. Fun. Not.

Anyway, I solved by resetting everything to factory defaults, moving away all gconf entries and .evolution/ files. Then it started to work again, of course then I needed to reconfigure it from scratch.

It turned out however that some old mail was only archived locally, and in a kind of weird format that looks like this:

$ ls -la Enrico/
total 336
drwx------ 2 enrico enrico   4096 Jul 23 03:05 .
drwxr-xr-x 7 enrico enrico   4096 Jul 23 03:12 ..
-rw------- 1 enrico enrico   3230 Dec  4  2010 113.HEADER
-rw------- 1 enrico enrico  14521 Dec  4  2010 113.TEXT
-rw------- 1 enrico enrico   3209 Oct 22  2010 134.HEADER
-rw------- 1 enrico enrico   2937 Oct 22  2010 134.TEXT
-rw------- 1 enrico enrico   3116 Jun 27  2011 15.
-rw------- 1 enrico enrico   3678 Jun 27  2011 168.
-rw------- 1 enrico enrico     73 Apr 27  2009 22.1.MIME
-rw------- 1 enrico enrico   3199 Apr 27  2009 22.2
-rw------- 1 enrico enrico     88 Apr 27  2009 22.2.MIME
[...]

I couldn't even find the name of that mail folder layout, let alone conversion tools. So I had to sit down and waste my sunday break writing software to convert that to a mbox file. Here's the tool, may it save you the awful time I had today: http://anonscm.debian.org/gitweb/?p=users/enrico/evo2mbox.git

Note: feel free to fork it, or send patches, but don't bother with feature requests. Evolution isn't and won't be a personal interest of mine. Anything that makes an afternoon at my parents more tiresome than a whole busy month of paid work, doesn't deserve to be.

Luckily they now seem to have changed the local folder format to Maildir.

Posted Mon Jul 23 03:27:50 2012 Tags:

Giving away distromatch

at last year's Fosdem I tried to inject a lot of energy into distromatch but shortly afterwards I've had to urgently rewrite the nm.debian.org website.

After Lars Wirzenius GTDFH talks in Bologna and Varese I wrote a tool which, among other things, is able to scan my home dir and list how many projects I'm working on.

The output was scary. Like, they are too many. Like, I couldn't even recite the list out of memory. And since I couldn't do that, I had no idea there were so many. And I kept being stressful because I couldn't manage to take care of them all properly.

Now that I became conscious of the situation, it's time to deal with it like a grown up, and politely back off from some of my irresponsible responsibilities.

Distromatch is one of them. It had just started as a proof of concept prototype, and I had the vision that it could be the basis for a fantastic culture of sharing and exchange of information across distributions.

I need to distinguish the vision from the responsibility. I still have that vision for distromatch, but I cannot take responsibility for making it happen.

So I am giving it up to anyone who has the time and resources to pick up that responsibility.

Current status

It works well enough as a prototype. I believe it can successfully map a large enough slice of packages, that one can prototype stuff based on it.

I have for example used it to export the Debtags categories for other distros, and the resulting file looked big enough to be used for prototyping category-based features on distributions that don't have them yet.

I think it also works well enough to support a few common use cases, like sharing screenshots, or doing most of the work of converting dependency lists from a distro to another.

And finally, anyone can deploy it, and work on it.

Existing data sources

Everything I index in the Debian distromatch deployment is available at http://dde.debian.net/exports/distromatch/. The rpm-based data in there comes from an export script I wrote that runs on Sophie, but which I cannot maintain properly.

This is an experimental export of Fedora and OpenSUSE data: http://tmp.vuntz.net/misc/distromatch/distromatch-opensuse-fedora.tar

All existing export scripts are found in distromatch git repo on gitorious.

Contacts I gathered at Fosdem

At Fosdem I devoted quite some work to get contacts from all possible distributions and software repositories, so that distromatch could be hooked into them. Here is a dump of what I have collected:

  • Debian: me
  • OpenSuse: Vincent Untz and Adrian Schröter
  • Fedora: Tom "Spot" Callaway
  • Arch: Tasser on IRC
  • CPAN: contact the people of https://metacpan.org/, on irc.perl.org:#metacpan or make an issue on github
  • NetBSD: ask on #netbsd on Freenode
  • FreeBSD: Baptiste Daroussin (bapt)
  • Mageia: Olivier Thauvin

Some of those contacts may have "expired" in the meantime: I wouldn't assume all of them still remember talking with me, although most probably still do.

My commitment for the time being

I am happy to commit, at the moment, to maintaining a working data export for Debian data. I can take responsibility for making it so that the Debian data for it stays up to date, and to fix it asap if it isn't the case.

I hope that now someone can take distromatch over from me, and make it grow to achieve its great potential.

Posted Sat Jul 21 16:54:18 2012 Tags:

More diversity in Debian skills

This blog post has been co-authored with Francesca Ciceri.

In his Debconf talk, zack said:

We need to understand how to invite people with different backgrounds than packaging to join the Debian project [...] I don't know what exactly, but we need to do more to attract those kinds of people.

Francesca and I know what we could do: make other kinds of contributions visible.

Basically, we should track and acknowledge the contributions of webmasters, translators, programmers, sysadmins, event organisers, and so on, at the same level as what we do for packagers: DDPO, minechangelogs, Portfolio...

For any non-packaging activity that we can make visible and credited, we get:

  • to acknowledge the people who do it, and show that they are active contributors in the project;

  • to acknowledge the work that gets done, and show the actual amount of non-packaging work that gets done in Debian every day;

  • to allow non-packagers to have a reputation, too: first of all, they deserve it, and among other things, it would make nm processing trivial.  

Here's an example: who's the lead translator for German? And if you are German, who's the lead translator for Spanish? Czech? Thai? I (Enrico) don't know the answers, not even for Italian, but we all should! Or at least it should be trivial to find out.

To start to change this, is just a matter of programming.

Francesca already worked on a list of trackable data sources, at least for translators.

Here are some more details, related to translation:

  • Translations can be tracked via the i18n robot (and relative statistics). This works only with teams who activated the robot and actively use the pseudo-urls in their messages on localisation mailing lists. Some translators don't bother to do it but it's ok to only support the main workflow. It beats extracting .po files from l10n-tagged BTS bugs at any rate.

  • DPN and website translations: for wml pages there's a specific field to be extracted for each translated page: grep for maintainer="name" on normal wml pages, while for DPN translations we have a specific translator="name" field. The problem is that this field is not mandatory, so sometimes there's no indication of the maintainer. Again, it's ok to only support the main workflow.

    Anyway, this is preferable to the cvs log: often the commit is done by the coordinator of the team and not by the actual translator. See above for the alternative solution of using the statistics provided by the i18n bot.

  • DDTSS: since the new release of DDTSS-Django, done by Martijn van Oosterhout about a year ago, the contributions are by default non-anonymous. This should be easy to track.

  • http://wiki.debian.org: it is more complicated because in the wiki we do not have a proper l10n translation workflow, so the only thing that can be tracked are changelogs $LANG/* pages. A nice idea would be to have translated pages list the version of the page that was translated and who did the translation.

  • translation of debian manuals and release notes: usually in the translation of manuals and long documentation there is a specific translator field.

And here are some notes about other fields:

  • DPN editors: for each issue there's a list of editors at the bottom of the page. In the wml: grep for editor=.

  • Artwork: artwork submitted via debianart are easy to track on the portal. Anyway usually you can find the author in the license and copyright file.

  • Programming: the only thing we have is the list of services which can be expanded if needed.

  • Press and publicity: there seems to be not much besides svn logs.

  • l10n-english: The Smith Review Project page has some tracking links. Other activities can probably only be tracked, at the moment, via mailing list activity.

  • Events: we can use the "main coordinator" field on www.debian.org/events/$year/$date-$eventname.wml: grep for <define-tag coord>; for events not published on the http://www.debian.org, but only on http://wiki.debian.org, the coordinator or the contact for the event is usually present on the page itself.

  • Sysadmins: we haven't asked DSA.

And finally, if you are still wondering who those translation coordinators are, they are listed here, although not all teams keep that page up to date.

Of course, when a data source is too hard to mine, it can make sense to see if the workflow could be improved, rather than spending months writing compicated mining code.

This is a fun project for people at Debconf to get together and try.

If by the end of the conference we had a way to credit some group of non-packaging contributors, even if just one like translators or website contributors, at least we would finally have started having official trackers for the activities of non-packagers.

Posted Thu Jul 12 14:01:54 2012 Tags:

Debtags for derivative distributions

Sometimes I do cool stuff and I forget to announce it.

Ok, so I recently announced a new Debtags website.

I forgot to say in the announcement that the new website does not only know of Debian packages: see for example this page, at the very bottom it says: "Distributions: oneiric, precise, sid, testing".

This means that already, here and now, debtags.debian.net can be used to tag packages from both Debian and Ubuntu, and can easily be extended to cover the entire Debian ecosystem.

If you are a package maintainer, you will notice that your maintainer page shows your packages from everywhere. If you want to filter things a bit, for example hide obsolete packages from an old Debian Stable or Ubuntu LTS, just click on the "Settings" link on the top right to configure the page.

How it works

The magic is in this mergepackages script, which is run daily, and exports merged Packages files at dde.debian.net. The debtags.debian.net concept of Packages and Sources files are just those all-merged.gz and all-merged-sources.gz.

The merging is simple: that rebuild script processes files in order, and the first version of a package that is found is chosen as the base for the one that will go in the merged Packages file. Some fields like "Description" are just taken from this pivot package, others like Architecture or dependencies are merged into it. It's arbitrary, but works for me: the result has all the packages with all their possible architectures and dependencies, and is ready to be indexed with apt-xapian-index.

At the moment I pull data from Debian and Ubuntu, but you can see that the script can easily be extended to pull data from any Debian-style ftp archive, so any Debian derivative can go in. I've already started negotiations with the Derivatives Census on how to add any Debian derivative and keep the list up to date.

How to export tags for your own distribution

I'll use Ubuntu as an example since the data is already available.

The way you add Debtags to the Ubuntu packages file is just this one:

  1. Get the full reviewed tag database
  2. Optionally filter out those packages that you are not interested in
  3. Tweak this script to build an overrides file.
  4. Give the overrides file to your favourite ftp archive building tool.

The make-overrides is a bit rusty: if you improve it, please send me your changes.

That is it, nothing else required, no excuses, it's ready, here, now!

Hitches and gotchas

This merged Packages file is a bit of a hack, and suffers from name conflicts across distributions, where two different softwares are packaged in two different distributions with the same name.

Ideally, name conflicts should not happen: if a derivative decided to package kate and call it gedit, they deserve to have it tagged uitoolkit::gtk. I think it's rather important that the whole Debian ecosystem works as much as possible with a single package namespace.

However, that reasoning fails if you take time into account: packages get renamed, like git and chromium, and may mean completely different things, for example, if you compare Debian Stable with Debian Sid.

This last is a problem caused by debtags only working with package names but not package versions. I have a strategy in mind based on being able to override the stable tag database using headers in debian/control; it still needs some details sorted out, but I'm confident we will be able to address these issues properly soon enough.

Why stop at the Debian ecosystem?

Why indeed. I'm clearly trying to use FOSDEM, and the CrossDistribution devroom as the venue to discuss just that.

Posted Fri Jan 20 15:12:33 2012 Tags:

Deploying distromatch

I have been working on allowing anyone to set up their own distromatch instance.

For Debian and Ubuntu, I can easily generate the distromatch input using UDD and the Contents files found in any mirrors.

For the whole RPM world, thanks to Olivier Thauvin I have been able to set up regular exports from the vast Sophie database.

I have set up distromatch access on DDE, which can also serve as a list of all working distributions so far. If you have access to the full dataset of package names and package contents for a distribution not in that list, please get in touch and we can add it.

I'm also exporting the full raw dataset which enables anyone to set up the same distromatch environment on their own machines.

Here is how:

# Get distromatch
git clone git://gitorious.org/appstream/distromatch.git
cd distromatch

# Fetch distribution information (updated every 2 days)
wget http://dde.debian.net/exports/distromatch-all.tar.gz

# Unpack it
mkdir data
tar -C data -zxf distromatch-all.tar.gz

# Reindex it (use --verbose if you are curious)
./distromatch --datadir=data --reindex --verbose

# Run it
./distromatch --datadir=data debian gedit

What does this mean? For example it means that if another distribution has some data (categories, screenshots...) that your distribution doesn't have, you can use distromatch to translate package names, then go and get it!

My next step is going to be to improve the distromatch functionality in DDE and possibly build a simple user friendly web interface to it. If you have some JQuery experience and would like to help, don't wait to get in touch.

Posted Fri Feb 18 13:46:30 2011 Tags:

update-apt-xapian-index on other distros

I've drafted a little HOWTO on using apt-xapian-index on non-Debian distributions.

The procedure has been tried on Mageia with some success, and there's no reason it wouldn't work everywhere else: the index itself does not depend on anything distro-specific.

Posted Tue Jan 25 23:01:45 2011 Tags:

Match package names across distributions

What would happen if we had a quick and reliable way to match package names across distributions?

These ideas came up at the appinstaller2011 meeting:

  • it would be easy to lookup screenshots in the local distro, and if there are none then fall back on other distributions;
  • it would be easy to port Debtags to other distributions, and possibly get changes back;
  • it would be trivial to add a [patches in $DISTRO] link to the PTS
  • it would be easy to point to other BTSes

We thought they were good ideas, so we started hacking.

To try it, you need to get the code and build the index first:

git clone git://git.debian.org/users/enrico/distromatch.git
cd distromatch
# Careful: 90Mb
wget http://people.debian.org/~enrico/dist-info.tar.gz
tar zxf dist-info.tar.gz
# Takes a long time to do the indexing
./distromatch --reindex --verbose

Then you can query it this way:

./distromatch $DISTRO $PKGNAME [$PKGNAME1 ...]

This would give you, for the package $PKGNAME in $DISTRO, the corresponding package names in all other distros for which we have data. If you do not provide package names, it automatically shows output for all packages in $DISTRO.

For example:

$ time ./distromatch debian libdigest-sha1-perl
debian:libdigest-sha1-perl fedora:perl-Digest-SHA1
debian:libdigest-sha1-perl mandriva:perl-Digest-SHA1
debian:libdigest-sha1-perl suse:perl-Digest-SHA1

real    0m0.073s
user    0m0.056s
sys 0m0.016s

Yes it's quick. It builds a Xapian index with the information it needs, and then it reuses it. As soon as I find a moment, I intend to deploy an instance of it on DDE.

It is using a range of different heuristics:

  • match packages by name;
  • match packages by desktop files contained within;
  • match packages by pkg-config metadata files contained within;
  • match packages by [/usr]/bin/* files contained within;
  • match packages by shared library files contained within;
  • match packages by devel library files contained within;
  • match packages by man pages contained within;
  • match stemmed form of development library package names;
  • match stemmed form of shared library package names;
  • match stemmed form of perl library package names;
  • match stemmed form of python library package names.

This list may get obsolete soon as more heuristics get implemented.

Euristics will never cover all corner cases we surely have, but the idea is that if we can match a sizable amout of packages, the rest can be somehow fixed by hand as needed.

The data it requires for a distribution should be rather straightforward to generate:

  1. a file which maps binary package names to source package names
  2. a file with the list of files in all the packages

For example:

$ ls -l dist-debian/
total 39688
-rw-r--r--  1 enrico enrico  1688249 Jan 20 17:37 binsrc
drwxr-xr-x  2 enrico enrico     4096 Jan 21 19:12 db
-rw-r--r--  1 enrico enrico 29960406 Jan 21 10:02 files.gz
-rw-r--r--  1 enrico enrico  8914771 Jan 21 18:39 interesting-files

$ head dist-debian/binsrc 
openoffice.org-dev openoffice.org
ext4-modules-2.6.32-5-4kc-malta-di linux-kernel-di-mipsel-2.6
linux-headers-2.6.30-2-common linux-2.6
libnspr4 nspr
ipfm ipfm
libforks-perl libforks-perl
med-physics debian-med
libntfs-3g-dev ntfs-3g
libguppi16 guppi
selinux selinux

$ zcat dist-debian/files.gz | head
memstat etc/memstat.conf
memstat usr/bin/memstat
memstat usr/share/doc/memstat/changelog.gz
memstat usr/share/doc/memstat/copyright
memstat usr/share/doc/memstat/memstat-tutorial.txt.gz
memstat usr/share/man/man1/memstat.1.gz
libdirectfb-dev usr/bin/directfb-config
libdirectfb-dev usr/bin/directfb-csource
libdirectfb-dev usr/include/directfb-internal/core/clipboard.h
libdirectfb-dev usr/include/directfb-internal/core/colorhash.h

interesting-files and db are generated when indexing.

To prove the usefulness of the idea (but does it need proving?), you can find in the same git repo a little example app (it took me 10 minutes to write it), that uses the distromatch engine to export Debtags tags to other distributions:

$ ./exportdebtags fedora | head
memstat: admin::benchmarking, interface::commandline, role::program, use::monitor
libdirectfb-dev: devel::lang:c, devel::library, implemented-in::c, interface::framebuffer, role::devel-lib
libkonqsidebarplugin4a: implemented-in::c++, role::shared-lib, suite::kde, uitoolkit::qt
libemail-simple-perl: devel::lang:perl, devel::library, implemented-in::perl, role::devel-lib, role::shared-lib, works-with::mail
libpoe-component-pluggable-perl: devel::lang:perl, devel::library, implemented-in::perl, role::shared-lib
manpages-ja: culture::japanese, made-of::man, role::documentation
libhippocanvas-dev: devel::library, qa::low-popcon, role::devel-lib
libexpat-ocaml-dev: devel::lang:ocaml, devel::library, implemented-in::c, implemented-in::ocaml, role::devel-lib, works-with-format::xml
libgnutls-dev: devel::library, role::devel-lib, suite::gnu

Just in case this made you itch to play with Debtags in a non-Debian distribution, I've generated the full datasets for Fedora, Mandriva and OpenSUSE.

Others have been working on the same matching problem. After we started writing code we started to become aware of existing work:

I'd like to make use of those efforts, maybe to cross-validate results, maybe even better as yet another heuristics.

Update:

I built a simple distromatch query system into DDE!

Posted Sat Jan 22 01:40:50 2011 Tags:

A prototype webby markety appy thing

What better way to introduce my work at an Application Installer meeting than to come with a prototype package browser modeled after shopping sites developed in just a few hours?

It's a little Flask webapp that just works on any Debian system, using the local apt-xapian-index as a backend. It has fast keyword search, faceted navigation and screenshots, and it runs on your system showing the packages that you have available.

Screenshot of packageshelf

To try it:

git clone git://git.debian.org/users/enrico/pkgshelf.git
cd pkgshelf
./web-server.py

Then visit http://localhost:5000

It didn't have much interface polishing, as it's just a quick technology demo. However you can see that:

  • keyword search is fast (fast enought that it could be made to search as you type);
  • relevant tags appear on the left, grouped by facets;
  • the most relevant tags are highlighted;
  • the less relevant tags could be hidden behind a [more] expander;
  • you can choose several strategies to hide packages you may find irrelevant.

Things that need doing:

  • hiding uninteresting facets;
  • making it pretty.

It's essentially JavaScript and CSS work. Anyone wants to play?

Posted Sat Jan 22 01:40:50 2011 Tags:

Cross-distro Meeting on Application Installer

I have been to a Cross-distro Meeting on Application Installer which to the best of our knowledge is also the first one of its kind. Credit goes to Vincent Untz for organising it, to OpenSUSE for hosting it and to the various sponsors for getting us there.

It went surprisingly well. We got along, got stuff done, did as much work as possible to agree on as many formats, protocols and technologies as we possibly could.

The timing of it is very important, as most major distros would like to adopt some of the features that just became popular in the various new app markets and stores, such as screenshots, user comments and ratings. It looks like a lot of new code is about to be written, or a lot of existing code is about to gain quite a bit of popularity.

For my part, I presented the work on Debtags and apt-xapian-index.

With regards to Debtags, other distros seem to be missing a compehensive classification system, and Debtags is, well, it.

With regards to apt-xapian-index, we just noticed that it's the perfect back-end for what everyone would like to do, and the index structure is rather distribution-agnostic, and it's been road-tested with considerable success by at least software-center, so it attracted quite a bit of interest, and will likely attract some more.

Just to prove a point I put together a prototype webby markety appy thing in just a few hours of work.

The meeting was also the ideal place to create a joint effort to match package names across distributions, which means that a lot of things that were hard to share before, such as screenshots, tags and patches, are suddenly not hard to share anymore.

Posted Sat Jan 22 01:40:50 2011 Tags:

fuss-launcher: an application launcher built on apt-xapian-index

Long ago I blogged about using apt-xapian-index to write an application launcher.

Now I just added a couple of new apt-xapian-index plugins that look like they have been made just for that.

In fact, they have indeed been made just for that.

After my blog post in 2008, people from Truelite and the FUSS project took up the challenge and wrote a launcher applet around my example engine.

The prototype has been quite successful in FUSS, and as a consequence I've been asked (and paid) to bring in some improvements.

The result, that I have just uploaded to NEW, is a package called fuss-launcher:

* New upstream release
   - Use newer apt-xapian-index: removed need of local index
   - Dragging a file in the launcher shows the applications that can open it
   - Remembers the applications launched more frequently
   - Allow to set a list of favourite applications

To get it:

  • apt-get install fuss-launcher (after it passed NEW);
  • or git clone http://git.fuss.bz.it/git/launcher.git/ and apt-get install python-gtk2 python-xapian python-xdg apt-xapian-index app-install-data

It requires apt-xapian-index >= 0.35.

To try it:

  1. Make sure your index is up to date, especially if you just installed app-install-data: just run update-apt-xapian-index as root.
  2. Run fuss-launcher.
  3. Click on the new tray icon to open the launcher dialog.
  4. Type some keywords and see the list of matching applications come to life as you type.

It's worth mentioning again that all this work was sponsored by Truelite and the Fuss project, which rocks.

Some screenshots:

When you open the launcher, by default it shows the most frequently started applicationss and the favourite applications:

launcher just opened

When you type some keywords, you get results as you type, and context-sensitive completion:

keyword search

When you drag a file on the launcher you only see the applications that can open that file:

drag files to the launcher

Posted Mon May 17 10:41:09 2010 Tags:
Posted Sat Jun 6 00:57:39 2009

Pages about OpenMoko.

Released nodm 0.7

I have released version 0.7 of nodm.

It only fixes one silly typo in autotools, which made it fail to build on Fedora.

Posted Sun May 23 21:36:52 2010 Tags:

Released nodm 0.6

I have released version 0.6 of nodm.

It is purely a bug fix release, trying harder to detect a console in order to get rid of a bug introduced with version 0.5

Posted Mon Aug 3 12:34:16 2009 Tags:

Released nodm 0.5

I have released version 0.5 of nodm.

New features:

  • truncate ~/.xsession-errors on startup: finally that file stops growing, and growing, and growing...
  • dynamic VT allocation: it can now avoid opening a virtual terminal if it is already in use.
Posted Fri Jul 24 02:29:55 2009 Tags:

Getting dbus signatures right from Vala

I am trying to play a bit with Vala on the FreeRunner.

The freesmartphone.org stack on the OpenMoko is heavily based on DBus. Using DBus from Vala is rather simple, if mostly undocumented: you get a few examples in the Vala wiki and you make do with those.

All works fine with simple methods. But what with providing callbacks to signals that have complex nested structures in their signatures, like aa{sv}? You try, and then if you don't get the method signature right, the signal is just silently not delivered because it does not match the method signature.

So this is how to provide a callback to org.freesmartphone.Usage.ResourceChanged, with signature sba{sv}:

public void on_resourcechanged(dynamic DBus.Object pos,
                   string name,
                   bool state,
                   HashTable<string, Value?> attributes)
{
    stderr.printf("Resource %s changed\n", name);
}

And this is how to provide a callback to org.freesmartphone.GPS.UBX.DebugPacket, with signature siaa{sv}:

protected void on_ubxdebug_packet(dynamic DBus.Object ubx, string clid, int length,
        HashTable<string, Value?>[] wrongdata)
{
    stderr.printf("Received UBX debug packet");

    // Ugly ugly work-around
    PtrArray< HashTable<string, Value?> >* data = (PtrArray< HashTable<string, Value?> >)wrongdata;

    stderr.printf("%u elements received", data->len);
}

What is happening here is that the only method signature that I found matching the dbus signature is this one. However, the unmarshaller for some reason gets it wrong, and passes a PtrArray instead of a HashTable array. So you need to cast it back to what you've actually been passed.

Figuring all this out took several long hours and was definitely not fun.

Posted Wed Jul 15 12:30:50 2009 Tags:

Mapping using the Openmoko FreeRunner headset

The FreeRunner has a headset which includes a microphone and a button. When doing OpenStreetMap mapping, it would be very useful to be able to keep tangogps on the display and be able to mark waypoints using the headset button, and to record an audio track using the headset microphone.

In this way, I can use tangogps to see where I need to go, where it's already mapped and where it isn't, and then I can use the headset to mark waypoints corresponding to the audio track, so that later I can take advantage of JOSM's audio mapping features.

Enter audiomap:

$ audiomap --help
Usage: audiomap [options]

Create a GPX and audio trackFind the times in the wav file when there is clear
voice among the noise

Options:
  --version      show program's version number and exit
  -h, --help     show this help message and exit
  -v, --verbose  verbose mode
  -m, --monitor  only keep the GPS on and monitor satellite status
  -l, --levels   only show input levels

If called without parameters, or with -v which is suggested, it will:

  1. Fix the mixer settings so that it can record from the headset and detect headset button presses.
  2. Show a monitor of GPS satellite information until it gets a fix.
  3. Synchronize the system time with the GPS time so that the timestamps of the files that are created afterwards are accurate.
  4. Start recording a GPX track.
  5. Start recording audio.
  6. Record a GPX waypoint for every headset button press.

When you are done, you stop audiomap with ^C and it will properly close the .wav file, close the tags in the GPX waypoint and track files and restore the mixer settings.

You can plug the headset out and record using the handset microphone, but then you will not be able to set waypoints until you plug the headset back in.

After you stop audiomap, you will have a track, waypoints and .wav file ready to be loaded in JOSM.

Big thanks go to Luca Capello for finding out how to detect headset button presses.

Posted Sun Jun 7 23:51:37 2009 Tags:

Simple tool to query the GPS using the OpenMoko FSO stack

I was missing a simple command line tool that allows me to perform basic GPS queries in shellscripts.

Enter getgps:

# getgps --help
Usage: getgps [options]

Simple GPS query tool for the FSO stack

Options:
  --version          show program's version number and exit
  -h, --help         show this help message and exit
  -v, --verbose      verbose mode
  -q, --quiet        suppress normal output
  --fix              check if we have a fix
  -s, --sync-time    set system time from GPS time
  --info             get all GPS information
  --info-connection  get GPS connection information
  --info-fix         get GPS fix information
  --info-position    get GPS position information
  --info-accuracy    get GPS accuracy information
  --info-course      get GPS course information
  --info-time        get GPS time information
  --info-satellite   get GPS satellite information

So finally I can write little GPS-aware scripts:

if getgps --fix -q
then
    start_gps_aware_program
else
    start_gps_normal_program
fi

Or this.

Posted Sun Jun 7 17:59:32 2009 Tags:

Voice-controlled waypoints

I have it in my TODO list to implement taking waypoints when pressing the headset button of the openmoko, but that is not done yet.

In the meantime, I did some experiments with audio mapping, and since I did not manage to enter waypoints while recording them, I was looking for a way to make use of them anyway.

Enter findvoice:

$ ./findvoice  --help
Usage: findvoice [options] wavfile

Find the times in the wav file when there is clear voice among the noise

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -v, --verbose         verbose mode
  -p NUM, --percentile=NUM
            percentile to use to discriminate noise from voice
            (default: 90)
  -t, --timestamps      print timestamps instead of human readable information

You give it a wav file, and it will output a list of timestamps corresponding to where it things that you were talking clearly and near the FreeRunner / voice recorder instead of leaving the recorder dangling to pick up background noise.

Its algorithm is crude and improvised because I have no background whatsoever in audio processing, but it basically finds those parts of the audio file where the variance of the samples is above a given percentile: the higher the percentile, the less timestamps you get; the lower the percentile, the more likely it is that it picks a period of louder noise.

For example, you can automatically extract waypoints out of an audio file by using it together with gpxinterpolate:

./findvoice -t today.wav | ./gpxinterpolate today.gpx > today-waypoints.gpx

The timestamps it outputs are computed using the modification time of the .wav file: if your system clock was decently synchronised (which you can do with getgps), then the mtime of the wav is the time of the end of the recording, which gives the needed reference to compute timestamps that are absolute in time.

For example:

getgps --sync-time
arecord file.wav
^C
./findvoice -t file.wav | ./gpxinterpolate today.gpx > today-waypoints.gpx
Posted Sun Jun 7 02:48:40 2009 Tags:

Geocoding Unix timestamps

Geocoding EXIF tags in JPEG images is fun, but there is more that can benefit from interpolating timestamps over a GPX track.

Enter gpxinterpolate:

$ ./gpxinterpolate --help
Usage: gpxinterpolate [options] gpxfile [gpxfile...]

Read one or more GPX files and a list of timestamps on standard input. Output
a GPX file with waypoints at the location of the GPX track at the given
timestamps.

Options:
  --version      show program's version number and exit
  -h, --help     show this help message and exit
  -v, --verbose  verbose mode

For example, you can create waypoints interpolating file modification times:

find . -printf "%Ts %p\n" | ./gpxinterpolate ~/tracks/*.gpx > myfiles.gpx

In case you wonder where you were when you modified or accessed a file, now you can find out.

Posted Sun Jun 7 02:07:43 2009 Tags:

Recording audio on the FreeRunner

The FreeRunner can record audio. It is nice to record audio: for example I can run the recording in background while I keep tangogps in the screen, and take audio notes about where I am while I am doing mapping for OpenStreetMap.

Here is the script that I put together to create geocoded audio notes:

#!/bin/sh

WORKDIR=~/rec
TMPINFO=`mktemp $WORKDIR/info.XXXXXXXX`

# Sync system time and get GPS info
echo "Synchronising system time..."
getgps --sync-time --info > $TMPINFO

# Compute an accurate basename for the files we generate
BASENAME=~/rec/rec-$(date +%Y-%m-%d-%H-%M-%S)
# Then give a proper name to the file with saved info
mv $TMPINFO $BASENAME.info

# Proper mixer settings for recording
echo "Recording..."
alsactl -f /usr/share/openmoko/scenarios/voip-handset.state restore
arecord -D hw -f cd -r 8000 -t wav $BASENAME.wav

echo "Done"

It works like this:

  1. It synchronizes the system time from the GPS (if there is a fix) so that the timestamps on the wav files will be as accurate as possible.
  2. It also gets all sort of information from the GPS and stores them into a file, should you want to inspect it later.
  3. It records audio until it gets interrupted.

The file name of the files that it generates corresponds to the beginning of the recording. The mtime of the wav file obviously corresponds to the end of the recording. This can be used to later georeference the start and end point of the recording.

You can use this to check mixer levels and that you're actually getting any input:

arecord -D hw -f cd -r 8000 -t wav -V mono /dev/null

The getgps script is now described in its own post.

You may now want to experiment, in JOSM, with "Preferences / Audio settings / Modified times (time stamps) of audio files".

Posted Sun Jun 7 01:30:37 2009 Tags:

Unpacking the new FreeRunner

I got myself a FreeRunner. Here are some notes from the first few days:

Available operating systems

The FreeRunner comes with the Om 2007.2 distribution: it works for basic phone things, and it has an opkg package manager that you can use to install all sort of extra software. Only issue: the SMS application was rather unstable with my SIM card.

I then tried Om 2008.8, which is a major new redesign, partly based on Qtopia. It's definitely more advanced on how it looks, with smooth animations all around, but I did not manage to get the GSM part to work, because I did not manage to get it to ask for a PIN.

I then tried plain Qtopia and I got what looks and feels like a properly working mobile phone. The only issue that it has is that I did not manage to make it suspend, so battery life is much shorter than it could be. Apparently, this should improve as kernel 2.6.26 reaches the phone. I kept Qtopia installed in the phone flash as a "stable" phone system.

And finally, Debian. Debian is based on the freesmartphone.org software stack, which is an attempt to create a UI agnostic DBUS frontend to the hardware that supports multiple applications running on top of it. It is a young project, but it can already drive a mobile phone in a useful way. It has a demo interface that is basically a showcase of what is implemented in the DBUS frontend, but does most of the basic things you need from a mobile phone, including taking and making phone calls. And then it has Debian behind, all of it. I need to buy a bigger microsd card.

Tips and tricks

Flashing things

  1. apt-get install dfu-util
  2. Start the Freerunner bios/bootloader, by holding down Power and then pressing AUX (standard bootloader), or by holding down AUX and then pressing Power (factory, unbrickable, read only fail safe bootloader).
  3. dfu-util -l shows you a list of devices it can access. If you see more than once, you need to specify in all dfu-util commands which one you want, using -d USBID. In my case, I have to always use dfu-util -d 0x1d50:0x5119
  4. To flash the kernel: dfu-util -a kernel -R -D /path/to/uImage
  5. To flash the root file system: dfu-util -a rootfs -R -D rootfs_filename.jffs2
  6. To flash the bootloader: dfu-util -a u-boot -R -D uboot_filename.bin
  7. To flash the u-boot configuration: dfu-util -a u-boot_env -D env.new

You can also download all of these things from the FreeRunner by using -U instead of -D.

Networking via USB

All of the distributions I tried, by default configure the USB as a gadget with ethernet over usb. You can form a lan with it using the cdc_ether module, and you will find your phone preconfigured as 192.168.0.202 expecting to find a gateway at 192.168.0.200.

I made myself this script to start and stop networking with the phone:

#!/bin/sh

case "$1" in
    start)
        iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE -s 192.168.0.200/29
        ifconfig usb0 192.168.0.200 netmask 255.255.255.248
        /etc/init.d/dnsmasq start
        echo 1 > /proc/sys/net/ipv4/ip_forward
        ;;
    stop)
        echo 0 > /proc/sys/net/ipv4/ip_forward
        /etc/init.d/dnsmasq stop
        iptables -t nat -F POSTROUTING
        ifconfig usb0 down
        ;;
esac

Besides doing masquerading, it also brings up dnsmasq so that the phone can always find a DNS together with the router.

Another useful trick is to configure the phone to share the approx cache with the laptop:

deb http://192.168.0.200:9999/debian unstable main
deb http://192.168.0.200:9999/debian experimental main
deb http://pkg-fso.alioth.debian.org/debian unstable main

So yes, my laptop can now be turned into a phone charger with networking, DNS and apt cache services. I shall look into hooking that script into dbus to have it run automatically when the phone is plugged and unplugged.

Changing the ringtone in Debian

In case you don't like the default ringtone, this is how to change it:

  1. vi /usr/share/python-support/fso-frameworkd/framework/subsystems/oeventd/receiver.py
  2. look for def _play( self ):
  3. change the codec in decoder = gst.element_factory_make, if needed. The list of available plugins is on the gstreamer website.
  4. change the path in filesrc.set_property
  5. /etc/init.d/fso-frameworkd restart and maybe /etc/init.d/zhone-session restart

Yes, the ringtone is currently hardcoded, but it does say it's a prototype after all. Or, if you prefer, it's fully configurable and the configuration can be found in /usr/share/python-support/fso-frameworkd.

Configuring the bootloader

You can connect to the u-boot bootloader via a serial terminal on /dev/ttyACM0. Type help and you will find that it can do a lot of things. The OpenMoko wiki has pages on the bootloader itself, its commands and the environment.

The environment is the configuration of the bootloader, similar somehow to /boot/grub/menu.lst. Unlike grub, you can edit the environment from within the bootloader and then save it using the saveenv command.

What I did:

  1. A boot entry for Debian, with the kernel on an ext2 partition instead of fat: setenv menu_3 Boot from microSD (FAT+ext2): setenv bootargs \${bootargs_base} rootfstype=ext2 root=/dev/mmcblk0p2 rootdelay=5 \${mtdparts} ro\; mmcinit\; fatload mmc 1 0x32000000 \${sd_image_name}\; bootm 0x32000000
  2. Stay in the bootloader until a choice has been made: setenv bootdelay -1
  3. Don't power down the bootloader when idle: setenv boot_menu_timeout 99999
  4. Always show the menu at boot: setenv stop_in_menu yes
  5. saveenv

So now my phone dual boots, and I can choose if I want a more reliable phone now (Qtopia) or if I want to play with my future phone (Debian).

Configuring ssh for two host keys on the same host

Minor issue, but annoying: since both QTopia and Debian show up on 192.168.0.202, ssh will complain about changed host keys. Here is how to configure ssh to avoid the problem (in ~/.ssh/config):

Host debian
HostName 192.168.0.202
User root
HostKeyAlias debian

Host qtopia
HostName 192.168.0.202
User root
HostKeyAlias qtopia
Posted Sat Jun 6 00:57:39 2009 Tags:
Posted Sat Jun 6 00:57:39 2009

Food and recipes.

Coppone e spinaci all'orientale

Ingredienti:

  • una bistecca di coppone
  • spinaci surgelati
  • olio
  • aglio
  • zenzero
  • peperoncino
  • anice stellato
  • salsa di soia
  • olio di sesamo tostato
  • pepe

Al supermercato hanno spesso delle bistecchine di coppone in sconto. Sono ottime sulla griglia, ma in mancanza di griglia una volta ho improvvisato questo, e ogni tanto lo rifaccio. È una cena velocissima che si può preparare quando non c'è niente in casa con ingredienti presi fuori dal freezer.

Soffriggere nell'olio l'aglio, lo zenzero e l'anice stellato.

Aggiungere il coppone tagliato a pezzetti e farlo rosolare. Mentre sta cuocendo, aggiungere il peperoncino sbriciolato, l'olio di sesamo e un po' di salsa di soia.

Quando la carne ha preso colore, aggiungere gli spinaci scongelati e rosolarli assieme alla carne e al suo sugo.

Regolare di sale con la salsa di soia e spolverare di pepe macinato prima di servire.

Posted Sat Mar 9 18:42:22 2013 Tags:

Besciamella al caffè e acciughe

Ingredienti:

Ispirato da un soufflé di broccoli al caffè mangiato alla meravigliosa trattoria Antichi Sapori a Parma, ho provato anch'io a combinare caffè e broccoli.

L'idea era fare una salsa da versare sui broccoli appena lessati. Lo chef Davide Sensi aveva parlato di caffè e acciughe, quindi ho deciso che il sapore della salsa dovrà venire da caffè e salsa di pesce thai. Per addensarli, ci potrebbe stare anche solo un classico roux.

Il risultato, una besciamella in cui il roux non è allungato con latte, ma con caffè, salsa di pesce thai e acqua di cottura dei broccoli.

La prima prova è venuta un po' troppo carica di caffè. Sui broccoli però ci sta benino.

Posted Fri Mar 8 19:17:04 2013 Tags:

Aubergine soup

I, too, have been guilty of discovering a shrivelled aubergine in the bottom of the fridge, and I think I improved on the recipe a bit.

First I softened the onion in butter, then I added the crushed garlic, the aubergine peeled and diced, and the cumin seeds. I let them all roast in the pan for a while, until the aubergines took some colour, then I added the stock. Carefully, as the pan with the roasting aubergines is far above 100°C and the first splash of water turns into steam very quickly.

When it was all soft and yummy, I added two spoonfuls of tahini, as a thickener, a generous grating of nutmeg, and blended the lot.

What came out is basically a soup version of baba ganoush, and it is yummy!

Posted Thu Nov 8 21:15:29 2012 Tags:

Spaghetti con friggitelli e mozzarella

Dosi per 4 persone:

  • 300 gr di spaghetti
  • 300 gr di friggitelli
  • 70 gr di mollica di pane
  • 125 gr di mozzarella
  • parmigiano grattugiato q.b.
  • 4 foglioline di menta
  • olio extra-vergine q.b.
  • sale & pepe

Laviamo i peperoncini, togliamo i semini e il picciolo, asciughiamoli e tagliamoli a striscette.

Tritiamo nel mixer la mollica di pane e doriamola in padella con 3 cucchiai di l'olio, finchè non diventerà croccanate e, mettiamolo da parte.

Tagliamo la mozzarella a dadini e teniamo anch'essa da parte.

Scaldiamo altri 5 cucchiai di olio, uniamo i peperoncini e facciamoli cuocere a fiamma viva per 10 minuti, regolandoli di sale e pepe.

Cuociamo la pasta, scoliamola e ripassiamola in padella con i peperoncini.

Facciamo saltare il tutto a fiamma vivace per qualche minuto.

Aggiungiamo la mozzarella a dadini, il pane croccante e il parmigiano.

Come tocco finale, uniamo la menta spezzettata e serviamo.

(via http://friggitelli.it/)

Fatta oggi, buona. Per le briciole di pane ho usato un avanzo della farcitura dei carciofi alla romana di ieri sera.

Con l'acqua di cottura dei carciofi, stasera risotto.

Posted Mon Apr 30 16:08:55 2012 Tags:

Risotto ai funghi (e un po' di banana)

Avevo voglia di sperimentare, e in casa avevo delle banane. Cosa ci si può fare, con delle banane?

Entra in gioco http://www.foodpairing.be/

Questo sito raggruppa vari ingredienti in base alla comunanza delle sostanze chimiche che gli danno il sapore. E chi c'è vicino alla banana? I FUNGHI!

Facciamo quindi un risotto coi funghi: solito fondo di cipolla soffritta nel burro finché non diventa trasparente, e poi giú un pezzetto di banana tagliato a pezzettini sottili, a soffriggere anche lui e a caramellarsi un po'. Infine, qualche pezzetto di porcino secco rinvenuto in acqua tiepida.

Aggiungiamo poi il riso, lasciamolo soffriggere anche lui un po' nell'intingolo, e poi allunghiamo col brodo (io avevo un dado apposta per il risotto ai funghi comprato nel vicino negozietto di cose belle).

Niente sale, pepe, burro per mantecare, niente. Una volta cotto, l'ho solo lasciato a riposare per 5 minuti.

Il risultato è stato delizioso. "Ci hai messo la panna? Come fa a essere cosí cremoso?". Saporito ma non dolce. E la banana si sente che c'è, ma non si sente che è banana.

Da oggi mi sa che nel risotto ai funghi il mio ingrediente segreto sarà un pezzetto di banana.

Da notare che, come si legge in http://khymos.org/pairings.php, la banana sta anche bene col prezzemolo, e cosí i funghi. Mi son scordato il prezzemolo nel risotto in questo esperimento, ma ci sarà nel prossimo: ce l'abbiamo anche fresco in giardino. La stessa pagina parla anche di un probabile abbinamento molecolare cacao-funghi... chissà.

Pagine collegate:

Posted Sat Jun 6 00:57:39 2009 Tags:

Fagottini di pollo agli spinaci

È da tempo che cerco di capire come cucinare una buona bistecca, e finalmente ho trovato un sito di cucina che parla la mia lingua.

Giochiamo quindi con la Reazione di Maillard. Dopo un discreto successo con una bistecchina da quattro soldi, è venuto il momento di cimentarsi col pollo, che è l'unica carne che piace alla morosa.

La carne di pollo ha proteine, ma non abbastanza zuccheri perché avvenga la reazione di Maillard. Ergo, mariniamo la carne in qualcosa che contenga zuccheri.

Guggolando "pollo" e "marinata", esce questa bella ricetta "Petti di pollo ripieni al miele e aceto balsamico". La ricetta dice: "rosolatevi il pollo (3-4 minuti per lato, a fuoco medio)", ma io Maillard lo volevo guardare negli occhi e "fuoco medio" non mi bastava, e poi in Inghilterra non si trova lo speck, e io in casa avevo degli spinaci e non dell'"insalata Tatsoi", ergo, ho pistolato la ricetta come mio solito:

Arrosto di pollo ripieno di spinaci

Ingredienti:

  • 3 petti di pollo a fette
  • 3 fette di bacon magro, senza cotenna (siamo in Inghilterra...)
  • 6-7 cubetti di spinaci surgelati
  • 2 cucchiai di miele (meglio se d’Acacia)
  • 2 cucchiai di aceto balsamico
  • 1 cucchiaio di salsa di soia
  • 1 cipollotto
  • peperoncino secco sminuzzato
  • olio, sale, pepe nero
  • vari spicchi di aglio

Ho scongelato gli spinaci in un tegamino a fuoco basso, assieme a 4 o 5 spicchietti d'aglio schiacciati.

Nel frattempo ho fatto la marinata con miele, aceto balsamico, salsa di soia, il cipollotto tagliato finemente, uno spicchio o due d'aglio schiacciato, il peperoncino, due cucchiai d'olio, sale, pepe.

Ho poi spiattellato un po' i petti di pollo, ci ho messo sopra una fetta di pancetta, poi ho intonacato con uno strato di spinaci, arrotolato il tutto e legato con lo spago.

Pronti i fagotti li ho messi a mollo nella marinata. Li ho lasciati lí una buona mezz'oretta poi li ho girati e li ho lasciati lí un'altra mezz'oretta, in modo che si impregnassero e colorassero bene.

A questo punto, ho dato la molla al forno a 180° (per dopo) e ho messo sul fuoco una padella (io ho usato il wok antiaderente) con un pochino d'olio d'oliva.

Mi sono assicurato di non far danni con la fiamma vivace: la reazione di Maillard avviene oltre i 140°, il punto di fumo dell'olio d'oliva è dai 190° ai 240°, e quello del teflon dell'antiaderente è di 300°, quindi i margini ci sono.

Fiamma alta, olio caldo, giú il primo fagotto di pollo: due minuti per lato, col tegame coperto per limitare i danni degli schizzi. Ogni fagotto fatto da entrambi i lati l'ho poi messo in una pirofila, ci ho versato sopra un filo d'olio e ho messo tutto in forno per una 20ina di minuti, per stare nel sicuro perché, seppure a fiamma alta, 2 minuti per lato non mi sembravano abbastanza per cuocere il pollo e il bacon all'interno.

Tra un fagotto e l'altro vale la pena togliere dal tegame il grosso dei fondi e metterlo da parte, altrimenti a star lí per 3 fagotti su 2 lati c'è il rischio che bruci. Alla fine, col tegame bello incrostato, ci ho versato del vino, ho aggiunto i fondi messi da parte, e col fuoco basso e il cucchiaio di legno ho scrostato il tutto. Ho poi aggiunto un po' di zucchero per contrastare l'aspro del vino e ho lasciato restringere, dopodiché ho filtrato col colino e ho ottenuto una salsina deliziosa da cospargenere sui fagotti al momento di servire.

Purtroppo non ho la foto perché, vuoi l'aspetto vuoi il profumo, tutti e tre i fagotti sono spariti prima che ci venisse in mente di fare la foto.

Siccome era rimasto dell'unto invitante nel fondo della pirofila e il forno era ancora caldo, ci ho poi arrostito delle patate al forno. In Italia ci saranno 40 gradi, ma qui si fa fatica ad arrivare a 20.

Il tutto, annaffiato da una bottiglia di dolcetto del monferrato che trovammo tempo fa in sconto al supermercato: saporito com'era il pollo, un vino bianco non avrebbe avuto speranza.

Posted Sat Jun 6 00:57:39 2009 Tags:
Posted Sat Jun 6 00:57:39 2009

Posts containing useful tips.

Resolving IP addresses in vim

A friend on IRC said: "I wish vim had a command to resolve all the IP addresses in a block of text".

But it does:

:<block>!perl -MSocket -pe 's/(\d+\.\d+\.\d+\.\d+)/gethostbyaddr(inet_aton($1), AF_INET)/ge'

If you use it often, put the perl command in a one-liner script and call it an editor macro. It works on other editors, too, and even without an editor at all. And it can be scripted!

We live with the power of Unix every day, so much that we risk forgetting how awesome it is.

Posted Wed Mar 7 14:07:07 2012 Tags:

SQLAlchemy, MySQL and sql_mode=traditional

As everyone should know, by default MySQL is an embarassing stupid toy:

mysql> create table foo (val integer not null);
Query OK, 0 rows affected (0.03 sec)

mysql> insert into foo values (1/0);
ERROR 1048 (23000): Column 'val' cannot be null

mysql> insert into foo values (1);
Query OK, 1 row affected (0.00 sec)

mysql> update foo set val=1/0 where val=1;
Query OK, 1 row affected, 1 warning (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 1

mysql> select * from foo;
+-----+
| val |
+-----+
|   0 |
+-----+
1 row in set (0.00 sec)

Luckily, you can tell it to stop being embarassingly stupid:

mysql> set sql_mode="traditional";
Query OK, 0 rows affected (0.00 sec)

mysql> update foo set val=1/0 where val=0;
ERROR 1365 (22012): Division by 0

(There is an even better sql mode you can choose, though: it is called "Install PostgreSQL")

Unfortunately, I've been hired to work on a project that relies on the embarassing stupid behaviour of MySQL, so I cannot set sql_mode=traditional globally or the existing house of cards will collapse.

Here is how you set it session-wide with SQLAlchemy 0.6.x: it took me quite a while to find out:

import sqlalchemy.interfaces

# Without this, MySQL will silently insert invalid values in the
# database, causing very long debugging sessions in the long run
class DontBeSilly(sqlalchemy.interfaces.PoolListener):
    def connect(self, dbapi_con, connection_record):
        cur = dbapi_con.cursor()
        cur.execute("SET SESSION sql_mode='TRADITIONAL'")
        cur = None
engine = create_engine(..., listeners=[DontBeSilly()])

Why does it take all that effort is beyond me. I'd have expected this to be turned on by default, possibly with a switch that insane people could use to turn it off.

Posted Mon Feb 27 19:45:58 2012 Tags:

Quei simpatici spammer di Aruba

Aruba ha deciso, di punto in bianco, di iscrivermi a tutte le loro newsletter.

Le newsletter non hanno link di deiscrizione. O meglio, forse ce l'hanno, ma si vedono solo decodificando la mail usando programmi che io non ho intenzione di usare. A prescindere dal link di deiscrizione, perché dovrei deiscrivermi da delle newsletter alle quali non mi sono mai iscritto?

Ho mandato questa mail a abuse@staff.aruba.it, e altre 3 segnalazioni dopo di questa, che ovviamente sono state ignorate:

Buon giorno,

vi segnalo questo spam inviato da voi stessi (in allegato la mail con
gli header intatti).

Potreste per favore procedere con provvedimenti disciplinari contro voi
stessi? Il vostro comportamento su internet viola le piú banali regole
di netiquette, ed è vostro interesse, come provider, istruire voi stessi
sulle stesse e farvele rispettare.


Cordiali saluti,

Enrico

Che dire, una nazione del terzo mondo si merita ISP da terzo mondo.

È pur sempre un'ottima scusa per studiarsi gli header_check di postfix: ora le mail delle newsletter di Aruba, che son tra l'altro dei patozzi da 300Kb l'una, incontrano un REJECT direttamente nella sessione SMTP:

550 5.7.1 Criminal third-world ISP spammers not accepted here.

Per farlo, ho aggiunto a /etc/postfix/main.cf:

# Reject aruba spam right away
header_checks = pcre:/etc/postfix/known_idiots.pcre

E poi ho creato il file /etc/postfix/known_idiots.pcre:

/^Received:.+smtpnewsletter[0-9]+.aruba.it/
REJECT Criminal third-world ISP spammers not accepted here.

Nel frattempo ho mandato un'email al Garante Privacy e una all'AGCOM, piú per curiosità che altro. Non mi aspetto nessuna risposta, ma se succede qualcosa lo aggiungo volentieri qui.

Posted Fri Oct 14 16:14:54 2011 Tags:

Award winning code

Me and Yuwei had a fun day at hhhmcr (#hhhmcr) and even managed to put together a prototype that won the first prize \o/

We played with the gmp24 dataset kindly extracted from Twitter by Michael Brunton-Spall of the Guardian into a convenient JSON dataset. The idea was to find ways of making it easier to look at the data and making sense of it.

This is the story of what we did, including the code we wrote.

The original dataset has several JSON files, so the first task was to put them all together:

#!/usr/bin/python

# Merge the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)

import simplejson
import os

res = []
for f in os.listdir("."):
    if not f.startswith("gmp24"): continue
    data = open(f).read().strip()
    if data == "[]": continue
    parsed = simplejson.loads(data)
    res.extend(parsed)

print simplejson.dumps(res)

The results however were not ordered by date, as GMP had to use several accounts to twit because Twitter was putting Greather Manchester Police into jail for generating too much traffic. There would be quite a bit to write about that, but let's stick to our work.

Here is code to sort the JSON data by time:

#!/usr/bin/python

# Sort the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)

import simplejson
import sys
import datetime as dt

all_recs = simplejson.load(sys.stdin)
all_recs.sort(key=lambda x: dt.datetime.strptime(x["created_at"], "%a %b %d %H:%M:%S +0000 %Y"))

simplejson.dump(all_recs, sys.stdout)

I then wanted to play with Tf-idf for extracting the most important words of every tweet:

#!/usr/bin/python

# tfifd - Annotate JSON elements with Tf-idf extracted keywords
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import sys, math
import simplejson
import re

# Read all the twits
records = simplejson.load(sys.stdin)

# All the twits by ID
byid = dict(((x["id"], x) for x in records))

# Stopwords we ignore
stopwords = set(["by", "it", "and", "of", "in", "a", "to"])

# Tokenising engine
re_num = re.compile(r"^\d+$")
re_word = re.compile(r"(\w+)")
def tokenise(tweet):
    "Extract tokens from a tweet"
    for tok in tweet["text"].split():
        tok = tok.strip().lower()
        if re_num.match(tok): continue
        mo = re_word.match(tok)
        if not mo: continue
        if mo.group(1) in stopwords: continue
        yield mo.group(1)

# Extract tokens from tweets
tokenised = dict(((x["id"], list(tokenise(x))) for x in records))

# Aggregate token counts
aggregated = {}
for d in byid.iterkeys():
    for t in tokenised[d]:
        if t in aggregated:
            aggregated[t] += 1
        else:
            aggregated[t] = 1

def tfidf(doc, tok):
    "Compute TFIDF score of a token in a document"
    return doc.count(tok) * math.log(float(len(byid)) / aggregated[tok])

# Annotate tweets with keywords
res = []
for name, tweet in byid.iteritems():
    doc = tokenised[name]
    keywords = sorted(set(doc), key=lambda tok: tfidf(doc, tok), reverse=True)[:5]
    tweet["keywords"] = keywords
    res.append(tweet)

simplejson.dump(res, sys.stdout)

I thought this was producing a nice summary of every tweet but nobody was particularly interested, so we moved on to adding categories to tweet.

Thanks to Yuwei who put together some useful keyword sets, we managed to annotate each tweet with a place name (i.e. "Stockport"), a social place name (i.e. "pub", "bank") and a social category (i.e. "man", "woman", "landlord"...)

The code is simple; the biggest work in it was the dictionary of keywords:

#!/usr/bin/python

# categorise - Annotate JSON elements with categories
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
# Copyright (C) 2010  Yuwei Lin <yuwei@ylin.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import sys, math
import simplejson
import re

# Electoral wards from http://en.wikipedia.org/wiki/List_of_electoral_wards_in_Greater_Manchester
placenames = ["Altrincham", "Sale West",
"Altrincham", "Ashton upon Mersey", "Bowdon", "Broadheath", "Hale Barns", "Hale Central", "St Mary", "Timperley", "Village",
"Ashton-under-Lyne",
"Ashton Hurst", "Ashton St Michael", "Ashton Waterloo", "Droylsden East", "Droylsden West", "Failsworth East", "Failsworth West", "St Peter",
"Blackley", "Broughton",
"Broughton", "Charlestown", "Cheetham", "Crumpsall", "Harpurhey", "Higher Blackley", "Kersal",
"Bolton North East",
"Astley Bridge", "Bradshaw", "Breightmet", "Bromley Cross", "Crompton", "Halliwell", "Tonge with the Haulgh",
"Bolton South East",
"Farnworth", "Great Lever", "Harper Green", "Hulton", "Kearsley", "Little Lever", "Darcy Lever", "Rumworth",
"Bolton West",
"Atherton", "Heaton", "Lostock", "Horwich", "Blackrod", "Horwich North East", "Smithills", "Westhoughton North", "Chew Moor", "Westhoughton South",
"Bury North",
"Church", "East", "Elton", "Moorside", "North Manor", "Ramsbottom", "Redvales", "Tottington",
"Bury South",
"Besses", "Holyrood", "Pilkington Park", "Radcliffe East", "Radcliffe North", "Radcliffe West", "St Mary", "Sedgley", "Unsworth",
"Cheadle",
"Bramhall North", "Bramhall South", "Cheadle", "Gatley", "Cheadle Hulme North", "Cheadle Hulme South", "Heald Green", "Stepping Hill",
"Denton", "Reddish",
"Audenshaw", "Denton North East", "Denton South", "Denton West", "Dukinfield", "Reddish North", "Reddish South",
"Hazel Grove",
"Bredbury", "Woodley", "Bredbury Green", "Romiley", "Hazel Grove", "Marple North", "Marple South", "Offerton",
"Heywood", "Middleton",
"Bamford", "Castleton", "East Middleton", "Hopwood Hall", "Norden", "North Heywood", "North Middleton", "South Middleton", "West Heywood", "West Middleton",
"Leigh",
"Astley Mosley Common", "Atherleigh", "Golborne", "Lowton West", "Leigh East", "Leigh South", "Leigh West", "Lowton East", "Tyldesley",
"Makerfield",
"Abram", "Ashton", "Bryn", "Hindley", "Hindley Green", "Orrell", "Winstanley", "Worsley Mesnes",
"Manchester Central",
"Ancoats", "Clayton", "Ardwick", "Bradford", "City Centre", "Hulme", "Miles Platting", "Newton Heath", "Moss Side", "Moston",
"Manchester", "Gorton",
"Fallowfield", "Gorton North", "Gorton South", "Levenshulme", "Longsight", "Rusholme", "Whalley Range",
"Manchester", "Withington",
"Burnage", "Chorlton", "Chorlton Park", "Didsbury East", "Didsbury West", "Old Moat", "Withington",
"Oldham East", "Saddleworth",
"Alexandra", "Crompton", "Saddleworth North", "Saddleworth South", "Saddleworth West", "Lees", "St James", "St Mary", "Shaw", "Waterhead",
"Oldham West", "Royton",
"Chadderton Central", "Chadderton North", "Chadderton South", "Coldhurst", "Hollinwood", "Medlock Vale", "Royton North", "Royton South", "Werneth",
"Rochdale",
"Balderstone", "Kirkholt", "Central Rochdale", "Healey", "Kingsway", "Littleborough Lakeside", "Milkstone", "Deeplish", "Milnrow", "Newhey", "Smallbridge", "Firgrove", "Spotland", "Falinge", "Wardle", "West Littleborough",
"Salford", "Eccles",
"Claremont", "Eccles", "Irwell Riverside", "Langworthy", "Ordsall", "Pendlebury", "Swinton North", "Swinton South", "Weaste", "Seedley",
"Stalybridge", "Hyde",
"Dukinfield Stalybridge", "Hyde Godley", "Hyde Newton", "Hyde Werneth", "Longdendale", "Mossley", "Stalybridge North", "Stalybridge South",
"Stockport",
"Brinnington", "Central", "Davenport", "Cale Green", "Edgeley", "Cheadle Heath", "Heatons North", "Heatons South", "Manor",
"Stretford", "Urmston",
"Bucklow-St Martins", "Clifford", "Davyhulme East", "Davyhulme West", "Flixton", "Gorse Hill", "Longford", "Stretford", "Urmston",
"Wigan",
"Aspull New Springs Whelley", "Douglas", "Ince", "Pemberton", "Shevington with Lower Ground", "Standish with Langtree", "Wigan Central", "Wigan West",
"Worsley", "Eccles South",
"Barton", "Boothstown", "Ellenbrook", "Cadishead", "Irlam", "Little Hulton", "Walkden North", "Walkden South", "Winton", "Worsley",
"Wythenshawe", "Sale East",
"Baguley", "Brooklands", "Northenden", "Priory", "Sale Moor", "Sharston", "Woodhouse Park"]

# Manual coding from Yuwei
placenames.extend(["City centre", "Tameside", "Oldham", "Bury", "Bolton",
"Trafford", "Pendleton", "New Moston", "Denton", "Eccles", "Leigh", "Benchill",
"Prestwich", "Sale", "Kearsley", ])
placenames.extend(["Trafford", "Bolton", "Stockport", "Levenshulme", "Gorton",
"Tameside", "Blackley", "City centre", "Airport", "South Manchester",
"Rochdale", "Chorlton", "Uppermill", "Castleton", "Stalybridge", "Ashton",
"Chadderton", "Bury", "Ancoats", "Whalley Range", "West Yorkshire",
"Fallowfield", "New Moston", "Denton", "Stretford", "Eccles", "Pendleton",
"Leigh", "Altrincham", "Sale", "Prestwich", "Kearsley", "Hulme", "Withington",
"Moss Side", "Milnrow", "outskirt of Manchester City Centre", "Newton Heath",
"Wythenshawe", "Mancunian Way", "M60", "A6", "Droylesden", "M56", "Timperley",
"Higher Ince", "Clayton", "Higher Blackley", "Lowton", "Droylsden",
"Partington", "Cheetham Hill", "Benchill", "Longsight", "Didsbury",
"Westhoughton"])


# Social categories from Yuwei
soccat = ["man", "woman", "men", "women", "youth", "teenager", "elderly",
"patient", "taxi driver", "neighbour", "male", "tenant", "landlord", "child",
"children", "immigrant", "female", "workmen", "boy", "girl", "foster parents",
"next of kin"]
for i in range(100):
    soccat.append("%d-year-old" % i)
    soccat.append("%d-years-old" % i)

# Types of social locations from Yuwei
socloc = ["car park", "park", "pub", "club", "shop", "premises", "bus stop",
"property", "credit card", "supermarket", "garden", "phone box", "theatre",
"toilet", "building site", "Crown court", "hard shoulder", "telephone kiosk",
"hotel", "restaurant", "cafe", "petrol station", "bank", "school",
"university"]


extras = { "placename": placenames, "soccat": soccat, "socloc": socloc }

# Normalise keyword lists
for k, v in extras.iteritems():
    # Remove duplicates
    v = list(set(v))
    # Sort by length
    v.sort(key=lambda x:len(x), reverse=True)

# Add keywords
def add_categories(tweet):
    text = tweet["text"].lower()
    for field, categories in extras.iteritems():
        for cat in categories:
            if cat.lower() in text:
                tweet[field] = cat
                break
    return tweet

# Read all the twits
records = (add_categories(x) for x in simplejson.load(sys.stdin))

simplejson.dump(list(records), sys.stdout)

All these scripts form a nice processing chain: each script takes a list of JSON records, adds some bit and passes it on.

In order to see what we have so far, here is a simple script to convert the JSON twits to CSV so they can be viewed in a spreadsheet:

#!/usr/bin/python

# Convert the JSON twits to CSV
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)

import simplejson
import sys
import csv

rows = ["id", "created_at", "text", "keywords", "placename"]

writer = csv.writer(sys.stdout)
for rec in simplejson.load(sys.stdin):
    rec["keywords"] = " ".join(rec["keywords"])
    rec["placename"] = rec.get("placename", "")
    writer.writerow([rec[row] for row in rows])

At this point we were coming up with lots of questions: "were there more reports on women or men?", "which place had most incidents?", "what were the incidents involving animals?"... Time to bring Xapian into play.

This script reads all the JSON tweets and builds a Xapian index with them:

#!/usr/bin/python

# toxapian - Index JSON tweets in Xapian
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import simplejson
import sys
import os, os.path
import xapian

DBNAME = sys.argv[1]

db = xapian.WritableDatabase(DBNAME, xapian.DB_CREATE_OR_OPEN)

stemmer = xapian.Stem("english")
indexer = xapian.TermGenerator()
indexer.set_stemmer(stemmer)
indexer.set_database(db)

data = simplejson.load(sys.stdin)
for rec in data:
    doc = xapian.Document()
    doc.set_data(str(rec["id"]))

    indexer.set_document(doc)
    indexer.index_text_without_positions(rec["text"])

    # Index categories as categories
    if "placename" in rec:
        doc.add_boolean_term("XP" + rec["placename"].lower())
    if "soccat" in rec:
        doc.add_boolean_term("XS" + rec["soccat"].lower())
    if "socloc" in rec:
        doc.add_boolean_term("XL" + rec["socloc"].lower())

    db.add_document(doc)

db.flush()

# Also save the whole dataset so we know where to find it later if we want to
# show the details of an entry
simplejson.dump(data, open(os.path.join(DBNAME, "all.json"), "w"))

And this is a simple command line tool to query to the database:

#!/usr/bin/python

# xgrep - Command line tool to query the GMP24 tweet Xapian database
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import simplejson
import sys
import os, os.path
import xapian

DBNAME = sys.argv[1]

db = xapian.Database(DBNAME)

stem = xapian.Stem("english")

qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
qp.add_boolean_prefix("place", "XP")
qp.add_boolean_prefix("soc", "XS")
qp.add_boolean_prefix("loc", "XL")

query = qp.parse_query(sys.argv[2],
    xapian.QueryParser.FLAG_BOOLEAN |
    xapian.QueryParser.FLAG_LOVEHATE |
    xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE |
    xapian.QueryParser.FLAG_WILDCARD |
    xapian.QueryParser.FLAG_PURE_NOT |
    xapian.QueryParser.FLAG_SPELLING_CORRECTION |
    xapian.QueryParser.FLAG_AUTO_SYNONYMS)

enquire = xapian.Enquire(db)
enquire.set_query(query)

count = 40
matches = enquire.get_mset(0, count)
estimated = matches.get_matches_estimated()
print "%d/%d results" % (matches.size(), estimated)

data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))

for m in matches:
    rec = data[m.document.get_data()]
    print rec["text"]

print "%d/%d results" % (matches.size(), matches.get_matches_estimated())

total = db.get_doccount()
estimated = matches.get_matches_estimated()
print "%d results over %d documents, %d%%" % (estimated, total, estimated * 100 / total)

Neat! Now that we have a proper index that supports all sort of cool things, like stemming, tag clouds, full text search with complex queries, lookup of similar documents, suggest keywords and so on, it was just fair to put together a web service to share it with other people at the event.

It helped that I had already written similar code for apt-xapian-index and dde before.

Here is the server, quickly built on bottle. The very last line starts the server and it is where you can configure the listening interface and port.

#!/usr/bin/python

# xserve - Make the GMP24 tweet Xapian database available on the web
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import bottle
from bottle import route, post
from cStringIO import StringIO
import cPickle as pickle
import simplejson
import sys
import os, os.path
import xapian
import urllib
import math

bottle.debug(True)

DBNAME = sys.argv[1]
QUERYLOG = os.path.join(DBNAME, "queries.txt")

data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))

prefixes = { "place": "XP", "soc": "XS", "loc": "XL" }
prefix_desc = { "place": "Place name", "soc": "Social category", "loc": "Social location" }

db = xapian.Database(DBNAME)

stem = xapian.Stem("english")

qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
for k, v in prefixes.iteritems():
    qp.add_boolean_prefix(k, v)

def make_query(qstring):
    return qp.parse_query(qstring,
        xapian.QueryParser.FLAG_BOOLEAN |
        xapian.QueryParser.FLAG_LOVEHATE |
        xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE |
        xapian.QueryParser.FLAG_WILDCARD |
        xapian.QueryParser.FLAG_PURE_NOT |
        xapian.QueryParser.FLAG_SPELLING_CORRECTION |
        xapian.QueryParser.FLAG_AUTO_SYNONYMS)


@route("/")
def index():
    query = urllib.unquote_plus(bottle.request.GET.get("q", ""))

    out = StringIO()
    print >>out, '''
<html>
<head>
<title>Query</title>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
<script type="text/javascript">
$(function(){
    $("#queryfield")[0].focus()
})
</script>
</head>
<body>
<h1>Search</h1>
<form method="POST" action="/query">
Keywords: <input type="text" name="query" value="%s" id="queryfield">
<input type="submit">
<a href="http://xapian.org/docs/queryparser.html">Help</a>
</form>''' % query

    print >>out, '''
<p>Example: "car place:wigan"</p>

<p>Available prefixes:</p>

<ul>
'''
    for pfx in prefixes.keys():
        print >>out, "<li><a href='/catinfo/%s'>%s - %s</a></li>" % (pfx, pfx, prefix_desc[pfx])
    print >>out, '''
</ul>
'''

    oldqueries = []
    if os.path.exists(QUERYLOG):
        total = db.get_doccount()
        fd = open(QUERYLOG, "r")
        while True:
            try:
                q = pickle.load(fd)
            except EOFError:
                break
            oldqueries.append(q)
        fd.close()

        def print_query(q):
            count = q["count"]
            print >>out, "<li><a href='/query?query=%s'>%s (%d/%d %.2f%%)</a></li>" % (urllib.quote_plus(q["q"]), q["q"], count, total, count * 100.0 / total)

        print >>out, "<p>Last 10 queries:</p><ul>"
        for q in oldqueries[:-10:-1]:
            print_query(q)
        print >>out, "</ul>"

        # Remove duplicates
        oldqueries = dict(((x["q"], x) for x in oldqueries)).values()

        print >>out, "<table>"
        print >>out, "<tr><th>10 queries with most results</th><th>10 queries with least results</th></tr>"
        print >>out, "<tr><td>"

        print >>out, "<ul>"
        oldqueries.sort(key=lambda x:x["count"], reverse=True)
        for q in oldqueries[:10]:
            print_query(q)
        print >>out, "</ul>"

        print >>out, "</td><td>"

        print >>out, "<ul>"
        nonempty = [x for x in oldqueries if x["count"] > 0]
        nonempty.sort(key=lambda x:x["count"])
        for q in nonempty[:10]:
            print_query(q)
        print >>out, "</ul>"

        print >>out, "</td></tr>"
        print >>out, "</table>"

    print >>out, '''
</body>
</html>'''
    return out.getvalue()

@route("/query")
@route("/query/")
@post("/query")
@post("/query/")
def query():
    query = bottle.request.POST.get("query", bottle.request.GET.get("query", ""))
    enquire = xapian.Enquire(db)
    enquire.set_query(make_query(query))

    count = 40
    matches = enquire.get_mset(0, count)
    estimated = matches.get_matches_estimated()
    total = db.get_doccount()

    out = StringIO()
    print >>out, '''
<html>
<head><title>Results</title></head>
<body>
<h1>Results for "<b>%s</b>"</h1>
''' % query

    if estimated == 0:
        print >>out, "No results found."
    else:
        # Give as results the first 30 documents; also use them as the key
        # ones to use to compute relevant terms
        rset = xapian.RSet()
        for m in enquire.get_mset(0, 30):
            rset.add_document(m.document.get_docid())

        # Compute the tag cloud
        class NonTagFilter(xapian.ExpandDecider):
            def __call__(self, term):
                return not term[0].isupper() and not term[0].isdigit()
        cloud = []
        maxscore = None
        for res in enquire.get_eset(40, rset, NonTagFilter()):
            # Normalise the score in the interval [0, 1]
            weight = math.log(res.weight)
            if maxscore == None: maxscore = weight
            tag = res.term
            cloud.append([tag, float(weight) / maxscore])
        max_weight = cloud[0][1]
        min_weight = cloud[-1][1]
        cloud.sort(key=lambda x:x[0])

        def mklink(query, term):
            return "/query?query=%s" % urllib.quote_plus(query + " and " + term)
        print >>out, "<h2>Tag cloud</h2>"
        print >>out, "<blockquote>"
        for term, weight in cloud:
            size = 100 + 100.0 * (weight - min_weight) / (max_weight - min_weight)
            print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(query, term), size, term)
        print >>out, "</blockquote>"

        print >>out, "<h2>Results</h2>"
        print >>out, "<p><a href='/'>Search again</a></p>"

        print >>out, "<p>%d results over %d documents, %.2f%%</p>" % (estimated, total, estimated * 100.0 / total)
        print >>out, "<p>%d/%d results</p>" % (matches.size(), estimated)

        print >>out, "<ul>"
        for m in matches:
            rec = data[m.document.get_data()]
            print >>out, "<li><a href='/item/%s'>%s</a></li>" % (rec["id"], rec["text"])
        print >>out, "</ul>"

        fd = open(QUERYLOG, "a")
        qinfo = dict(q=query, count=estimated)
        pickle.dump(qinfo, fd)
        fd.close()

    print >>out, '''
<a href="/">Search again</a>

</body>
</html>'''
    return out.getvalue()

@route("/item/:id")
@route("/item/:id/")
def show(id):
    rec = data[id]

    out = StringIO()
    print >>out, '''
<html>
<head><title>Result %s</title></head>
<body>
<h1>Raw JSON record for twit %s</h1>
<pre>''' % (rec["id"], rec["id"])

    print >>out, simplejson.dumps(rec, indent=" ")

    print >>out, '''
</pre>
</body>
</html>'''
    return out.getvalue()

@route("/catinfo/:name")
@route("/catinfo/:name/")
def catinfo(name):
    prefix = prefixes[name]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Values for %s</title></head>
<body>
''' % name

    terms = [(x.term[len(prefix):], db.get_termfreq(x.term)) for x in db.allterms(prefix)]
    terms.sort(key=lambda x:x[1], reverse=True)
    freq_min = terms[0][1]
    freq_max = terms[-1][1]

    def mklink(name, term):
        return "/query?query=%s" % urllib.quote_plus(name + ":" + term)

    # Build tag cloud
    print >>out, "<h1>Tag cloud</h1>"
    print >>out, "<blockquote>"
    for term, freq in sorted(terms[:20], key=lambda x:x[0]):
        size = 100 + 100.0 * (freq - freq_min) / (freq_max - freq_min)
        print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(name, term), size, term)
    print >>out, "</blockquote>"

    print >>out, "<h1>All terms</h1>"
    print >>out, "<table>"
    print >>out, "<tr><th>Occurrences</th><th>Name</th></tr>"
    for term, freq in terms:
        print >>out, "<tr><td>%d</td><td><a href='/query?query=%s'>%s</a></td></tr>" % (freq, urllib.quote_plus(name + ":" + term), term)
    print >>out, "</table>"

    print >>out, '''
</body>
</html>'''
    return out.getvalue()

# Change here for bind host and port
bottle.run(host="0.0.0.0", port=8024)

...and then we presented our work and ended up winning the contest.

This was the story of how we wrote this set of award winning code.

Posted Sat Oct 16 01:36:08 2010 Tags:

Computing time offsets between EXIF and GPS

I like the idea of matching photos to GPS traces. In Debian there is gpscorrelate but it's almost unusable to me because of bug #473362 and it has an awkward way of specifying time offsets.

Here at SoTM10 someone told me that exiftool gained -geosync and -geotag options. So it's just a matter of creating a little tool that shows a photo and asks you to type the GPS time you see in it.

Apparently there are no bindings or GIR files for gtkimageview in Debian, so I'll have to use C.

Here is a C prototype:

/*
 * gpsoffset - Compute EXIF time offset from a photo of a gps display
 *
 * Use with exiftool -geosync=... -geotag trace.gpx DIR
 *
 * Copyright (C) 2009--2010  Enrico Zini <enrico@enricozini.org>
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program; if not, write to the Free Software
 * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 */


#define _XOPEN_SOURCE /* glibc2 needs this */
#include <time.h>
#include <gtkimageview/gtkimageview.h>
#include <libexif/exif-data.h>
#include <stdio.h>
#include <stdlib.h>

static int load_time(const char* fname, struct tm* tm)
{
    ExifData* exif_data = exif_data_new_from_file(fname);
    ExifEntry* exif_time = exif_data_get_entry(exif_data, EXIF_TAG_DATE_TIME);
    if (exif_time == NULL)
    {
        fprintf(stderr, "Cannot find EXIF timetamp\n");
        return -1;
    }

    char buf[1024];
    exif_entry_get_value(exif_time, buf, 1024);
    //printf("val2: %s\n", exif_entry_get_value(t2, buf, 1024));

    if (strptime(buf, "%Y:%m:%d %H:%M:%S", tm) == NULL)
    {
        fprintf(stderr, "Cannot match EXIF timetamp\n");
        return -1;
    }

    return 0;
}

static time_t exif_ts;
static GtkWidget* res_lbl;

void date_entry_changed(GtkEditable *editable, gpointer user_data)
{
    const gchar* text = gtk_entry_get_text(GTK_ENTRY(editable));
    struct tm parsed;
    if (strptime(text, "%Y-%m-%d %H:%M:%S", &parsed) == NULL)
    {
        gtk_label_set_text(GTK_LABEL(res_lbl), "Please enter a date as YYYY-MM-DD HH:MM:SS");
    } else {
        time_t img_ts = mktime(&parsed);
        int c;
        int res;
        if (exif_ts < img_ts)
        {
            c = '+';
            res = img_ts - exif_ts;
        }
        else
        {
            c = '-';
            res = exif_ts - img_ts;
        }
        char buf[1024];
        if (res > 3600)
            snprintf(buf, 1024, "Result: %c%ds -geosync=%c%d:%02d:%02d",
                    c, res, c, res / 3600, (res / 60) % 60, res % 60);
        else if (res > 60)
            snprintf(buf, 1024, "Result: %c%ds -geosync=%c%02d:%02d",
                    c, res, c, (res / 60) % 60, res % 60);
        else 
            snprintf(buf, 1024, "Result: %c%ds -geosync=%c%d",
                    c, res, c, res);
        gtk_label_set_text(GTK_LABEL(res_lbl), buf);
    }
}

int main (int argc, char *argv[])
{
    // Work in UTC to avoid mktime applying DST or timezones
    setenv("TZ", "UTC");

    const char* filename = "/home/enrico/web-eddie/galleries/2010/04-05-Uppermill/P1080932.jpg";

    gtk_init (&argc, &argv);

    struct tm exif_time;
    if (load_time(filename, &exif_time) != 0)
        return 1;

    printf("EXIF time: %s\n", asctime(&exif_time));
    exif_ts = mktime(&exif_time);

    GtkWidget* window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
    GtkWidget* vb = gtk_vbox_new(FALSE, 0);
    GtkWidget* hb = gtk_hbox_new(FALSE, 0);
    GtkWidget* lbl = gtk_label_new("Timestamp:");
    GtkWidget* exif_lbl;
    {
        char buf[1024];
        strftime(buf, 1024, "EXIF time: %Y-%m-%d %H:%M:%S", &exif_time);
        exif_lbl = gtk_label_new(buf);
    }
    GtkWidget* date_ent = gtk_entry_new();
    res_lbl = gtk_label_new("Result:");
    GtkWidget* view = gtk_image_view_new();
    GdkPixbuf* pixbuf = gdk_pixbuf_new_from_file(filename, NULL);

    gtk_box_pack_start(GTK_BOX(hb), lbl, FALSE, TRUE, 0);
    gtk_box_pack_start(GTK_BOX(hb), date_ent, TRUE, TRUE, 0);

    gtk_signal_connect(GTK_OBJECT(date_ent), "changed", (GCallback)date_entry_changed, NULL);
    {
        char buf[1024];
        strftime(buf, 1024, "%Y-%m-%d %H:%M:%S", &exif_time);
        gtk_entry_set_text(GTK_ENTRY(date_ent), buf);
    }

    gtk_widget_set_size_request(view, 500, 400);
    gtk_image_view_set_pixbuf(GTK_IMAGE_VIEW(view), pixbuf, TRUE);
    gtk_container_add(GTK_CONTAINER(window), vb);
    gtk_box_pack_start(GTK_BOX(vb), view, TRUE, TRUE, 0);
    gtk_box_pack_start(GTK_BOX(vb), hb, FALSE, TRUE, 0);
    gtk_box_pack_start(GTK_BOX(vb), exif_lbl, FALSE, TRUE, 0);
    gtk_box_pack_start(GTK_BOX(vb), res_lbl, FALSE, TRUE, 0);
    gtk_widget_show_all(window);

    gtk_main ();

    return 0;
}

And here is its simple makefile:

CFLAGS=$(shell pkg-config --cflags gtkimageview libexif)
LDFLAGS=$(shell pkg-config --libs gtkimageview libexif)

gpsoffset: gpsoffset.c

It's a simple prototype but it's a working prototype and seems to do the job for me.

I currently cannot find out why after I click on the text box, there seems to be no way to give the focus back to the image viewer so I can control it with keys.

There is another nice algorithm to compute time offsets to be implemented: you choose a photo taken from a known place and drag it on that place on a map: you can then look for the nearest point on your GPX trace and compute the time offset from that.

I have seen that there are programs for geotagging photos that implement all such algorithms, and have a nice UI, but I haven't seen any in Debian.

Are there any such softwares that can be packaged?

If not, the interpolation and annotation tasks can now already be performed by exiftool, so it's just a matter of building a good UI, and I would love to see someone picking up the task.

Posted Sun Jul 11 12:34:04 2010 Tags:

Searching OSM nodes in Spatialite

Third step of my SoTM10 pet project: finding the POIs.

I put together a query to find all nodes with a given tag inside a bounding box, and also a query to find all the tag values for a given tag name inside a bounding box.

The result is this simple POI search engine:

#
# poisearch - simple geographical POI search engine
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#

from pysqlite2 import dbapi2 as sqlite

class PoiDB(object):
    def __init__(self):
        self.db = sqlite.connect("pois.db")
        self.db.enable_load_extension(True)
        self.db.execute("SELECT load_extension('libspatialite.so')")
        self.oldsearch = []
        self.bbox = None

    def set_bbox(self, xmin, xmax, ymin, ymax):
        '''Set bbox for searches'''
        self.bbox = (xmin, xmax, ymin, ymax)

    def tagid(self, name, val):
        '''Get the database ID for a tag'''
        c = self.db.cursor()
        c.execute("SELECT id FROM tag WHERE name=? AND value=?", (name, val))
        res = None
        for row in c:
            res = row[0]
        return res

    def tagnames(self):
        '''Get all tag names'''
        c = self.db.cursor()
        c.execute("SELECT DISTINCT name FROM tag ORDER BY name")
        for row in c:
            yield row[0]

    def tagvalues(self, name, use_bbox=False):
        '''
        Get all tag values for a given tag name,
        optionally in the current bounding box
        '''
        c = self.db.cursor()
        if self.bbox is None or not use_bbox:
            c.execute("SELECT DISTINCT value FROM tag WHERE name=? ORDER BY value", (name,))
        else:
            c.execute("SELECT DISTINCT tag.value FROM poi, poitag, tag"
                      " WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE ("
                      "       xmin >= ? AND xmax <= ? AND ymin >= ? AND ymax <= ?) )"
                      "   AND poitag.tag = tag.id AND poitag.poi = poi.id"
                      "   AND tag.name=?",
                      self.bbox + (name,))
        for row in c:
            yield row[0]

    def search(self, name, val):
        '''Get all name:val tags in the current bounding box'''
        # First resolve the tagid
        tagid = self.tagid(name, val)
        if tagid is None: return

        c = self.db.cursor()
        c.execute("SELECT poi.name, poi.data, X(poi.geom), Y(poi.geom) FROM poi, poitag"
                  " WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE ("
                  "       xmin >= ? AND xmax <= ? AND ymin >= ? AND ymax <= ?) )"
                  "   AND poitag.tag = ? AND poitag.poi = poi.id",
                  self.bbox + (tagid,))
        self.oldsearch = []
        for row in c:
            self.oldsearch.append(row)
            yield row[0], simplejson.loads(row[1]), row[2], row[3]

    def count(self, name, val):
        '''Count all name:val tags in the current bounding box'''
        # First resolve the tagid
        tagid = self.tagid(name, val)
        if tagid is None: return

        c = self.db.cursor()
        c.execute("SELECT COUNT(*) FROM poi, poitag"
                  " WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE ("
                  "       xmin >= ? AND xmax <= ? AND ymin >= ? AND ymax <= ?) )"
                  "   AND poitag.tag = ? AND poitag.poi = poi.id",
                  self.bbox + (tagid,))
        for row in c:
            return row[0]

    def replay(self):
        for row in self.oldsearch:
            yield row[0], simplejson.loads(row[1]), row[2], row[3]

Problem 3 solved: now on to the next step, building a user interface for it.

Posted Sat Jul 10 15:50:31 2010 Tags:

Importing OSM nodes into Spatialite

Second step of my SoTM10 pet project: creating a searchable database with the points. What a fantastic opportunity to learn Spatialite.

Learning Spatialite is easy. For example, you can use the two tutorials with catchy titles that assume your best wish in life is to create databases out of shapefiles using a pre-built, i386-only executable GUI binary downloaded over an insecure HTTP connection.

To be fair, the second of those tutorials is called "An almost Idiot's Guide", thus expliciting the requirement of being an almost idiot in order to happily acquire and run software in that way.

Alternatively, you can use A quick tutorial to SpatiaLite which is so quick it has examples that lead you to write SQL queries that trigger all sorts of vague exceptions at insert time. But at least it brought me a long way forward, at which point I could just cross reference things with PostGIS documentation to find out the right way of doing things.

So, here's the importer script, which will probably become my reference example for how to get started with Spatialite, and how to use Spatialite from Python:

#!/usr/bin/python

#
# poiimport - import nodes from OSM into a spatialite DB
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#

import xml.sax
import xml.sax.handler
from pysqlite2 import dbapi2 as sqlite
import simplejson
import sys
import os

class OSMPOIReader(xml.sax.handler.ContentHandler):
    '''
    Filter SAX events in a OSM XML file to keep only nodes with names
    '''
    def __init__(self, consumer):
        self.consumer = consumer

    def startElement(self, name, attrs):
        if name == "node":
            self.attrs = attrs
            self.tags = dict()
        elif name == "tag":
            self.tags[attrs["k"]] = attrs["v"]

    def endElement(self, name):
        if name == "node":
            lat = float(self.attrs["lat"])
            lon = float(self.attrs["lon"])
            id = int(self.attrs["id"])
            #dt = parse(self.attrs["timestamp"])
            uid = self.attrs.get("uid", None)
            uid = int(uid) if uid is not None else None
            user = self.attrs.get("user", None)

            self.consumer(lat, lon, id, self.tags, user=user, uid=uid)

class Importer(object):
    '''
    Create the spatialite database and populate it
    '''
    TAG_WHITELIST = set(["amenity", "shop", "tourism", "place"])

    def __init__(self, filename):
        self.db = sqlite.connect(filename)
        self.db.enable_load_extension(True)
        self.db.execute("SELECT load_extension('libspatialite.so')")
        self.db.execute("SELECT InitSpatialMetaData()")
        self.db.execute("INSERT INTO spatial_ref_sys (srid, auth_name, auth_srid,"
                        " ref_sys_name, proj4text) VALUES (4326, 'epsg', 4326,"
                        " 'WGS 84', '+proj=longlat +ellps=WGS84 +datum=WGS84"
                        " +no_defs')")
        self.db.execute("CREATE TABLE poi (id int not null unique primary key,"
                        " name char, data text)")
        self.db.execute("SELECT AddGeometryColumn('poi', 'geom', 4326, 'POINT', 2)")
        self.db.execute("SELECT CreateSpatialIndex('poi', 'geom')")
        self.db.execute("CREATE TABLE tag (id integer primary key autoincrement,"
                        " name char, value char)")
        self.db.execute("CREATE UNIQUE INDEX tagidx ON tag (name, value)")
        self.db.execute("CREATE TABLE poitag (poi int not null, tag int not null)")
        self.db.execute("CREATE UNIQUE INDEX poitagidx ON poitag (poi, tag)")
        self.tagid_cache = dict()

    def tagid(self, k, v):
        key = (k, v)
        res = self.tagid_cache.get(key, None)
        if res is None:
            c = self.db.cursor()
            c.execute("SELECT id FROM tag WHERE name=? AND value=?", key)
            for row in c:
                self.tagid_cache[key] = row[0]
                return row[0]
            self.db.execute("INSERT INTO tag (id, name, value) VALUES (NULL, ?, ?)", key)
            c.execute("SELECT last_insert_rowid()")
            for row in c:
                res = row[0]
            self.tagid_cache[key] = res
        return res

    def __call__(self, lat, lon, id, tags, user=None, uid=None):
        # Acquire tag IDs
        tagids = []
        for k, v in tags.iteritems():
            if k not in self.TAG_WHITELIST: continue
            for val in v.split(";"):
                tagids.append(self.tagid(k, val))

        # Skip elements that don't have the tags we want
        if not tagids: return

        geom = "POINT(%f %f)" % (lon, lat)
        self.db.execute("INSERT INTO poi (id, geom, name, data)"
                        "     VALUES (?, GeomFromText(?, 4326), ?, ?)", 
                (id, geom, tags["name"], simplejson.dumps(tags)))

        for tid in tagids:
            self.db.execute("INSERT INTO poitag (poi, tag) VALUES (?, ?)", (id, tid))


    def done(self):
        self.db.commit()

# Get the output file name
filename = sys.argv[1]

# Ensure we start from scratch
if os.path.exists(filename):
    print >>sys.stderr, filename, "already exists"
    sys.exit(1)

# Import
parser = xml.sax.make_parser()
importer = Importer(filename)
handler = OSMPOIReader(importer)
parser.setContentHandler(handler)
parser.parse(sys.stdin)
importer.done()

Let's run it:

$ ./poiimport pois.db < pois.osm 
SpatiaLite version ..: 2.4.0    Supported Extensions:
        - 'VirtualShape'        [direct Shapefile access]
        - 'VirtualDbf'          [direct Dbf access]
        - 'VirtualText'         [direct CSV/TXT access]
        - 'VirtualNetwork'      [Dijkstra shortest path]
        - 'RTree'               [Spatial Index - R*Tree]
        - 'MbrCache'            [Spatial Index - MBR cache]
        - 'VirtualFDO'          [FDO-OGR interoperability]
        - 'SpatiaLite'          [Spatial SQL - OGC]
PROJ.4 Rel. 4.7.1, 23 September 2009
GEOS version 3.2.0-CAPI-1.6.0
$ ls -l --si pois*
-rw-r--r-- 1 enrico enrico 17M Jul  9 23:44 pois.db
-rw-r--r-- 1 enrico enrico 37M Jul  9 16:20 pois.osm
$ spatialite pois.db
SpatiaLite version ..: 2.4.0    Supported Extensions:
        - 'VirtualShape'        [direct Shapefile access]
        - 'VirtualDbf'          [direct DBF access]
        - 'VirtualText'         [direct CSV/TXT access]
        - 'VirtualNetwork'      [Dijkstra shortest path]
        - 'RTree'               [Spatial Index - R*Tree]
        - 'MbrCache'            [Spatial Index - MBR cache]
        - 'VirtualFDO'          [FDO-OGR interoperability]
        - 'SpatiaLite'          [Spatial SQL - OGC]
PROJ.4 version ......: Rel. 4.7.1, 23 September 2009
GEOS version ........: 3.2.0-CAPI-1.6.0
SQLite version ......: 3.6.23.1
Enter ".help" for instructions
spatialite> select id from tag where name="amenity" and value="fountain";
24
spatialite> SELECT poi.name, poi.data, X(poi.geom), Y(poi.geom) FROM poi, poitag WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE (xmin >= 2.56 AND xmax <= 2.90 AND ymin >= 41.84 AND ymax <= 42.00) ) AND poitag.tag = 24 AND poitag.poi = poi.id;
Font Picant de la Cellera|{"amenity": "fountain", "name": "Font Picant de la Cellera"}|2.616045|41.952449
Font de Can Pla|{"amenity": "fountain", "name": "Font de Can Pla"}|2.622354|41.974724
Font de Can Ribes|{"amenity": "fountain", "name": "Font de Can Ribes"}|2.62311|41.979193

It's impressive: I've got all sort of useful information for the whole of Spain in just 17Mb!

Let's put it to practice: I'm thirsty, is there any water fountain nearby?

spatialite> SELECT count(1) FROM poi, poitag WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE (xmin >= 2.80 AND xmax <= 2.85 AND ymin >= 41.97 AND ymax <= 42.00) ) AND poitag.tag = 24 AND poitag.poi = poi.id;
0

Ouch! No water fountains mapped in Girona... yet.

Problem 2 solved: now on to the next step, trying to show the results in some usable way.

Posted Sat Jul 10 09:10:35 2010 Tags:

Filtering nodes out of OSM files

I have a pet project here at SoTM10: create a tool for searching nearby POIs while offline.

The idea is to have something in my pocket (FreeRunner or N900), which doesn't require an internet connection, and which can point me at the nearest fountains, post offices, atm machines, bars and so on.

The first step is to obtain a list of POIs.

In theory one can use Xapi but all the known Xapi servers appear to be down at the moment.

Another attempt is to obtain it by filtering all nodes with the tags we want out of a planet OSM extract. I downloaded the Spanish one and set to work.

First I tried with xmlstarlet, but it ate all the RAM and crashed my laptop, because for some reason, on my laptop the Linux kernels up to 2.6.32 (don't now about later ones) like to swap out ALL running apps to cache I/O operations, which mean that heavy I/O operations swap out the very programs performing them, so the system gets caught in some infinite I/O loop and dies. Or at least this is what I've figured out so far.

So, we need SAX. I put together this prototype in Python, which can process a nice 8MB/s of OSM data for quite some time with a constant, low RAM usage:

#!/usr/bin/python

#
# poifilter - extract interesting nodes from OSM XML files
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#


import xml.sax
import xml.sax.handler
import xml.sax.saxutils
import sys

class XMLSAXFilter(xml.sax.handler.ContentHandler):
    '''
    A SAX filter that is a ContentHandler.

    There is xml.sax.saxutils.XMLFilterBase in the standard library but it is
    undocumented, and most of the examples using it you find online are wrong.
    You can look at its source code, and at that point you find out that it is
    an offensive practical joke.
    '''
    def __init__(self, downstream):
        self.downstream = downstream

    # ContentHandler methods

    def setDocumentLocator(self, locator):
        self.downstream.setDocumentLocator(locator)

    def startDocument(self):
        self.downstream.startDocument()

    def endDocument(self):
        self.downstream.endDocument()

    def startPrefixMapping(self, prefix, uri):
        self.downstream.startPrefixMapping(prefix, uri)

    def endPrefixMapping(self, prefix):
        self.downstream.endPrefixMapping(prefix)

    def startElement(self, name, attrs):
        self.downstream.startElement(name, attrs)

    def endElement(self, name):
        self.downstream.endElement(name)

    def startElementNS(self, name, qname, attrs):
        self.downstream.startElementNS(name, qname, attrs)

    def endElementNS(self, name, qname):
        self.downstream.endElementNS(name, qname)

    def characters(self, content):
        self.downstream.characters(content)

    def ignorableWhitespace(self, chars):
        self.downstream.ignorableWhitespace(chars)

    def processingInstruction(self, target, data):
        self.downstream.processingInstruction(target, data)

    def skippedEntity(self, name):
        self.downstream.skippedEntity(name)

class OSMPOIHandler(XMLSAXFilter):
    '''
    Filter SAX events in a OSM XML file to keep only nodes with names
    '''
    PASSTHROUGH = ["osm", "bound"]
    TAG_WHITELIST = set(["amenity", "shop", "tourism", "place"])

    def startElement(self, name, attrs):
        if name in self.PASSTHROUGH:
            self.downstream.startElement(name, attrs)
        elif name == "node":
            self.attrs = attrs
            self.tags = []
            self.propagate = False
        elif name == "tag":
            if self.tags is not None:
                self.tags.append(attrs)
                if attrs["k"] in self.TAG_WHITELIST:
                    self.propagate = True
        else:
            self.tags = None
            self.attrs = None

    def endElement(self, name):
        if name in self.PASSTHROUGH:
            self.downstream.endElement(name)
        elif name == "node":
            if self.propagate:
                self.downstream.startElement("node", self.attrs)
                for attrs in self.tags:
                    self.downstream.startElement("tag", attrs)
                    self.downstream.endElement("tag")
                self.downstream.endElement("node")

    def ignorableWhitespace(self, chars):
        pass

    def characters(self, content):
        pass

# Simple stdin->stdout XMl filter
parser = xml.sax.make_parser()
handler = OSMPOIHandler(xml.sax.saxutils.XMLGenerator(sys.stdout, "utf-8"))
parser.setContentHandler(handler)
parser.parse(sys.stdin)

Let's run it:

$ bzcat /store/osm/spain.osm.bz2 | pv | ./poifilter > pois.osm
[...]
$ ls -l --si pois.osm
-rw-r--r-- 1 enrico enrico 19M Jul 10 23:56 pois.osm
$ xmlstarlet val pois.osm 
pois.osm - valid

Problem 1 solved: now on to the next step: importing the nodes in a database.

Posted Fri Jul 9 16:28:15 2010 Tags:

Tweaking locale settings

I sometimes meet some Italian programmer who prefers his system to be in English, so they get untranslated manpages and error messages.

I sometimes notice that their solution often leaves them something to complain about.

Some set LANG=C and complain they can't see accented characters.

Some set LANG=en_US.UTF-8 and complain that OpenOffice Calc wants dates in the format MM/DD/YYYY which is an Abomination unto Nuggan, as well as unto Me.

But the locales system can do much better than that. In fact, most times people would be extremely happy with LC_MESSAGES in English, and everything else in Italian:

$ locale
LANG=it_IT.UTF-8
LC_CTYPE="it_IT.UTF-8"
LC_NUMERIC="it_IT.UTF-8"
LC_TIME="it_IT.UTF-8"
LC_COLLATE="it_IT.UTF-8"
LC_MONETARY="it_IT.UTF-8"
LC_MESSAGES=en_US.UTF-8
LC_PAPER="it_IT.UTF-8"
LC_NAME="it_IT.UTF-8"
LC_ADDRESS="it_IT.UTF-8"
LC_TELEPHONE="it_IT.UTF-8"
LC_MEASUREMENT="it_IT.UTF-8"
LC_IDENTIFICATION="it_IT.UTF-8"
LC_ALL=

A way to do this (is there a better one?) is to tell the display manager (GDM, KDM...) to use the Italian locale, and then override the right LC_* bits in ~/.xsessionrc:

$ cat ~/.xsessionrc
export LC_MESSAGES=en_US.UTF-8

That does the trick: English messages, with Proper currency, Proper dates, accented letters sort Properly, Proper A4 printer paper, Proper SI units. Even Nuggan would be happy.

Posted Sat Apr 17 15:39:19 2010 Tags:

Temporarily disabling file caching

Does it happen to you that you cp a big, big file (say, similar in order of magnitude to the amount of RAM) and the system becomes rather unusable?

It looks like Linux is saying "let's cache this", and as you copy it will try to free more and more ram in order to cache the big file you're copying. In the end, all the RAM is full with file data that you are not going to need.

This varies according to how /proc/sys/vm/swappiness is set.

I learnt about posix_fadvise and I tried to play with it. The result is this preloadable library that hooks into open(2) and fadvises everything as POSIX_FADV_DONTNEED.

It is all rather awkward. fadvise in that way will discard existing cache pages if the file is already cached, which is too much. Ideally one would like to say "don't cache this because of me" without stepping on the feet of other system activities.

Also, I found I need to also hook into write(2) and run fadvise after every single write, because you can't fadvise a file to be written in its entirety, unless you pass fadvise the file size in advance. But the size of the output file cannot be known by the preloaded library, so meh.

So, now I can run: nocache cp bigfile someplace/ without trashing the existing caches. I can also run nocache tar zxf foo.tar.gz and so on. I wish, of course, that there were no need to do so in the first place.

Here is the nocache library source code, for reference:

/*
 * nocache - LD_PRELOAD library to fadvise written files to not be cached
 *
 * Copyright (C) 2009--2010 Enrico Zini <enrico@enricozini.org>
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program; if not, write to the Free Software
 * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 */

#define _XOPEN_SOURCE 600
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <dlfcn.h>
#include <stdarg.h>
#include <errno.h>
#include <stdio.h>

typedef int (*open_t)(const char*, int, ...);
typedef int (*write_t)(int fd, const void *buf, size_t count);

int open(const char *pathname, int flags, ...)
{
    static open_t func = 0;
    int res;
    if (!func)
        func = (open_t)dlsym(RTLD_NEXT, "open");

    // Note: I wanted to add O_DIRECT, but it imposes restriction on buffer
    // alignment
    if (flags & O_CREAT)
    {
        va_list ap;
        va_start(ap, flags);
        mode_t mode = va_arg(ap, mode_t);
        res = func(pathname, flags, mode);
        va_end(ap);
    } else
        res = func(pathname, flags);

    if (res >= 0)
    {
        int saved_errno = errno;
        int z = posix_fadvise(res, 0, 0, POSIX_FADV_DONTNEED);
        if (z != 0) fprintf(stderr, "Cannot fadvise on %s: %m\n", pathname);
        errno = saved_errno;
    }

    return res;
}

int write(int fd, const void *buf, size_t count)
{
    static write_t func = 0;
    int res;
    if (!func)
        func = (write_t)dlsym(RTLD_NEXT, "write");

    res = func(fd, buf, count);

    if (res > 0)
    {
        int saved_errno = errno;
        int z = posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED);
        if (z != 0) fprintf(stderr, "Cannot fadvise during write: %m\n");
        errno = saved_errno;
    }

    return res;
}

Updates

Steve Schnepp writes:

Robert Love did a O_STREAMING patch for 2.4. It wasn't merged in 2.6 since POSIX_FADV_NOREUSE should be used instead.

But unfortunatly it's currently mapped either as WILLNEED or as a noop.

It seems that there is a google code project that has spawned to control this.

Posted Mon Mar 8 13:26:28 2010 Tags:
Posted Sat Jun 6 00:57:39 2009

Cazzeggio.

Non importa che mi dai del voi

Dai, non importa che mi dai del voi

In che senso?

Eh, mi dici sempre "voi informatici", "voi tecnici", "voi..."

Posted Fri Dec 19 15:55:20 2014 Tags:

Spelling a chilometri zero

Lo spelling internazionale è troppo globalizzato, e volete recuperare un attimo la dimensione del posto dove siete nati e cresciuti?

Da oggi c'è questo script che fa per voi: gli dite dove abitate, e lui vi crea lo spelling a chilometri zero.

$ git clone git@gitorious.org:trespolo/osmspell.git
$ cd osmspell
$ ./osmspell "San Giorgio di Piano"
1: San Giorgio di Piano, BO, EMR, Italia
2: San Giorgio di Piano, Via Codronchi, San Giorgio di Piano, BO, EMR, Italia
3: San Giorgio Di Piano, Via Libertà, San Giorgio di Piano, BO, EMR, Italia
Choose one: 1
Center: 44.6465332, 11.3790398
A Argelato, Altedo
B Bentivoglio, Bologna, Boschi
C Cinquanta, Castagnolo Minore, Castel Maggiore, Cento
D Dosso
E Eremo di Tizzano
F Funo di Argelato, Finale Emilia, Ferrara, Fiesso
G Gherghenzano, Galliera, Gesso
I Il Cucco, Irnerio, Idice
L Località Fortuna, Lovoleto, Lippo
M Malacappa, Massumatico, Minerbio, Marano
N Navile
O Osteriola, Ozzano dell'Emilia, Oca
P Piombino, Padulle, Poggio Renatico, Piave
Q Quarto Inferiore, Quattrina
R Rubizzano, Renazzo, Riale
S San Giorgio di Piano, Saletto
T Torre Verde, Tintoria, Tombe
U Uccellino
V Venezzano Mascarino, Vigarano Mainarda, Veduro
X XII Morelli
Z Zenerigolo, Zola Predosa

I dati vengono da OSM, e lo script è un ottimo esempio di come usarne la API di geolocazione (veloci) e la API di query geografica (lenta).

Posted Sat Jan 4 00:38:16 2014 Tags:

Poesia: "Lavatrice"

Pensavo fosse pail,

invece ora è feltro.

Posted Tue Dec 3 22:32:23 2013 Tags:

Shops

Christmas songs should only ever be played on Christmas day.

In church.

At midnight.

Unless I happen to be there.

Posted Mon Dec 2 14:07:58 2013 Tags:

Airports

Photo of a commercial in London City airport saying 'In the lap of luxury - Want to reach a captive audience with dwell time? Why advertise anywhere else? - London City Airport Media Sales'

In the airport, we are not travellers. We are a captive audience with dwell time.

In other words, suckers stuck in a room where the only pastime provided is spending money and staring at advertisements selling advertisement space in rooms full of suckers like them.

Posted Fri Nov 22 18:58:00 2013 Tags:

Explanation of umarell

Umarell /uma'rɛl/ (oo-mah-rell), n; pl. Umarells. People in a community who offer all sorts of comments to those who are trying to get some work done, but who are not doing any work themselves.

Etymology and further details

Umarell is a word that entered Italian slang in Bologna and is spreading to nearby towns, occasionally even across Italy. It comes from the Bolognese for "cute/odd little man".

"Umarells" are those people, usually retired men, who spend time watching construction works, often holding their hands behind their back, occasionally commenting on what is going on, sometimes trying to tell the workers what to do.

It's easy to find examples on the internet; the word was popularised by a blog collecting photos, which has even been published into a book.

With some Italian Debian friends, we realised that umarell is the perfect word to describe those people in a community, who offer all sorts of comments to those who are trying to get some work done, but who are not doing any work themselves.

I think that it is a word that fits perfectly, and since I'm likely going to use it blissfully anywhere, here is a page that temporarily explains what it means until the Oxford English Dictionary picks it up.

Posted Fri Sep 20 13:27:07 2013 Tags:

Yet another Ubuntu anecdote

Some posts on planet made me remember of a little Canonical-related story of mine.

Many years ago I shortly contracted for Canonical. It was interesting and fun.

At the time I didn't have any experience of being temporarily hired by a foreign company, so I rang my labour union to get an appointment, to make sure with them that everything was allright.

The phone call went more or less like this:

Me:

Hello. I have received this contract for temporary employment by a foreign company and I wondered if I could book an appointment to come show it to you to see if it's all ok.

Their answer rather cut me short:

Hi. Be careful! People get temporary employment from obscure companies with the headquarters, like, in the Isle of Man, they do the job, the company disappears and they never get paid. There's bad stuff out there!

I looked at the contract, the heading said something like "Canonical ltd, Douglas, Isle of Man".

I was certain that the union people would have never understood what was going on. I politely thanked them for their time and hung up. However, to this day I still regret that I didn't insist:

Uh, yes, the company is indeed in the Isle of Man. But what if I told you that it's owned by an astronaut?

I just signed the contract and had a good time.

Posted Sat Jan 15 10:35:36 2011 Tags:

Mailman defaults

Monopoly Chance: It's the first of the month. / You're flooded with mailman junk / Skip a turn.

Posted Fri Oct 1 12:03:53 2010 Tags:

My rule to see if a framework is worth of attention

I came up with a little rule:

In order to be worth of any attention, a framework must be stable enough that I can charge money to train people to use it.

This probably applies to other kinds of software stacks, libraries, development environments and, well, to most software applications.

In the context of python web frameworks, this means that:

  • If it changes API all the time it is not worth of attention, because my customers won't get value for their money, as they'd continuously need retraining and rewriting their software.
  • If I see lots of DeprecationWarnings it is not worth of attention, because my customers will see them and blame me for teaching them deprecated stuff.
  • If fixes for bugs affecting the stable version are only distributed "in a recent git" or "in the next development version", and they are not backported into a new bugfix-only stable release, then it is not worth of attention, because:
    • my customers' business is to develop their own products based on the framework.
    • My customers' business is not to be maintaning in-house stable updates of the framework. Although if the framework's community is nice enough they might end up giving a hand.
  • If it requires virtualenv or can only be obtained through easy_install it is not worth of attention, because:
    • my customers are not interested in maintaning custom deployment environments over time.
    • My customers are not interested in tracking each and every single library's upstream development to keep their production system free of bugs.
    • My customers are used to getting software through a proper distribution which also takes care of security updates.
    • I am paid to teach them how to use a framework, not a custom python-only package management system.
    • In my experience, if distributions have trouble keeping packages up to date, upstream is doing something fundamentally wrong.

In light of this rule, I regret to notice that I see very few python web frameworks worth of any attention.

Posted Wed Aug 4 15:32:24 2010 Tags:

On python stable APIs

There is a theory which states that if ever anyone discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable.

There is another theory which states that this has already happened.

In Debian testing:

/usr/lib/python2.6/dist-packages/sqlalchemy/types.py:547: SADeprecationWarning: The Binary type has been renamed to LargeBinary.

In Debian Lenny:

ImportError: cannot import name LargeBinary

I was starting to think that SQLAlchemy wasn't too bad, since I've been using it for 6 months and I haven't seen its API change yet.

But there it is, a beautiful reminder that SQLAlchemy, too, is part of the marvelously autistic Python ecosystem.

Posted Mon Jul 19 16:14:25 2010 Tags:
Posted Sat Jun 6 00:57:39 2009
pdo

Pages exported to http://planet.debian.org.

If you happen to know a browser developer...

Do you happen to know a developer of Firefox or Chrome or some other mainstream browser?

If so, can you please talk to them about our experiments with Client Certificate authentication in Debian?

Client Certificate authentication rocks; with just a couple of little tweaks in the interface, it would be pretty close to perfect.

Visiting sites without using a certificate

If I want to browse a site unauthenticated instead of using a certificate, at the moment I can hit "Cancel" on the certificate popup menu, and it works nicely. I feel quite confused when I do that, though, because it's not clear to me if I am canceling use of certificates, or canceling the visit to the site.

Can you please change the wording on the Cancel button to something more descriptive?

See/change current certificate selection

My top wish is, once I choise to use (or not use) a certificate for a site, to be able to see which certificate I'm using and possibly change it.

At the moment I did not find a way to see what certificate I'm using, and the browser will remember the choice until it gets closed and reopened.

At the moment I can use a Private or Incognito window to switch identities or to stop authenticated access and continue anonymously, and that helps me immensely.

I think however that the ultimate solution could be to have the https lockpad popup show an indication of what certificate is currently being used, and offer a way to re-trigger certificate selection. That would be so cool.

Also, once the certificate choice can be seen and changed at any time, it could just get remembered so that sites can be visited again without any prompts, even after the browser has been closed and reopened. That would be, to me, the ultimate convenience.

Thanks! <3

Thank you very much for all the work you have already put into this: I have been told that a few years ago using client certificate was unthinkable, and now it seems to be down to just a couple of papercuts. And SPKAC/keygen seriously rocks!

I have been constantly impressed by how well this all works right now.

Posted Tue Sep 1 17:25:04 2015 Tags:

My semi serious stand up comedy notes

Video

Disclaimers

“Someone has said that it requires less mental effort to condemn than to think.”

(Emma Goldman, on several things including mailing list flamewars)

Fascinating Aïda's "Dogging" song.

Look for "dogging etiquette" for more examples of code of conducts. Just don't take your computer for repair immediately afterwards™.

Introduction

Every daring attempt to make a great change in existing conditions, every lofty vision of new possibilities for the human race, has been labeled Utopian.

(Emma Goldman, on the Debian Social Contract)

I am going to talk about many topics that we all know have so much in common:

  • Anarchism
  • Poliamory
  • BDSM
  • and Free Software

They are all, after all:

  • People
  • Consensually
  • Doing Things
  • Together

BDSM

A person is no less a slave because they are allowed to choose a new master once in a term of years.

(Lysander Spooner about proprietary cloud service providers)

If you thought you've seen it all with recursive acronyms, here's a chain acronym: Bondage Discipline, Dominance Submission, Sado Masochism.

Why I think BDSM is interesting: not (just) because of whips, but for having a lot of awareness about power releationships. Why should one accept from a coworker a level of abuse that would be considered a hard limit when negotiating with a trusted dom?

The BDSM Free Software definition: "I refuse to be bound by software I cannot negotiate with".

YKINMKBYKIOK (Your Kink Is Not My Kink But Your Kink Is Okay) is a nice example of dealing with diversity, and it also definitely solves the emacs vs vi debate.

Comfort zones, safewords, traffic light flow control, safety.

"No means no", unless there has been a long discussion first, and a safeword is in place, in which case "Fuzzy purple unicorn" means "no"

"No means no", and if someone insists after a "no", it becomes harassment.

"No means no" is a precondition for being able to say "yes": http://pervocracy.blogspot.de/2011/03/no-and-no-and-no-and-yes.html

Aftercare! Aftercare! Release parties! High fives! Solidarity after flamewars or votes!

Poliamory

If love does not know how to give and take without restrictions, it is not love, but a transaction that never fails to lay stress on a plus and a minus.

(Emma Goldman, on volunteer projects)

Polyamory is the practice, desire, or acceptance of intimate relationships that are not exclusive with respect to other sexual or intimate relationships, with knowledge and consent of everyone involved.

Compersion, n: the feeling you get when someone else also takes good care of one of your packages.

We currently allow only one value in the Maintainer field: * takeover is traumatic, because values can only be replaced * if values could be added instead, and removed when they don't make sense anymore...

What is your definition of love? My current one is: my world is better with you in it.

Relationship anarchy is the practice of forming relationships which are not bound by rules aside from what the people involved mutually agree on. How do you call a relationship that is bound by rules that the people involved do not agree on?

From discussions after the talk

New Relationship Energy, the excitement when you start to maintain a new package, and the risk of been carried away by the excitement and neglecting all the other ones.

Consent

Anarchism, to me, means not only the denial of authority, not only a new economy, but a revision of the principles of morality. It means the development of the individual as well as the assertion of the individual. It means self-responsibility, and not leader worship.

(Voltairine de Cleyre about trusting lintian warnings)

You need to know what you are doing, and what situation you're putting yourself into.

You need to know that the person asking a question really is able to accept any answer, and take it seriously.

You need to feel that you have alternatives.

Be selfish when you ask, honest when you reply, and when others reply, take them seriously. If any of this doesn't stand, I find it hard to trust that we are in a consensual situation.

When is one supposed to learn about consent?

  • I see little consensuality in standard education.
  • I see little consensuality at work.

Consent explained with tea.

Practical advice

Anarchism has but one infallible, unchangeable motto, ‘Freedom.’ Freedom to discover any truth, freedom to develop, to live naturally and fully.

(Lucy Parsons about the DFSG)

Relationship advice and work advice have a lot in common:

Relationship advice from 99 ways to ruin an open source project

Online participation advice from How to Screw Up Your Relationship (and make everyone miserable while you’re at it)

Packaging advice from BDSM Basics: 20 Unsolicited Tips for New Dominants

Advice about joining a new community from Advice to a newbie submissive about dominants

♥ ♥ ♥

Dear Debian, and dear everyone contributing to it: my world is better with you in it.

I love you all :* <3

Posted Tue Aug 18 11:48:08 2015 Tags:

Expectations and needs

All people ever say is: "thank you" (a celebration of life) and "please" (an opportunity to make life more wonderful). (Marshall Rosenberg)

Sometimes, when I see the word "expectation" I try to read it as "need" and see how things change.

I noticed that this tends to reframe situations in a way that makes me feel more comfortable.

I noticed that I tend to instinctively perceive "expectations" as "do this or there will be consequences", and I tend to instinctively perceive "needs" as "do this if you want to see me happy".

I noticed that my motivation to care for someone's expectations tend to be something close to fear, and my motivation to care for someone's needs tends to be something close to love.

This might give me a bit more hints on The art of asking: I will not expect you to do something for me, I'll just allow myself to be loved, liked or helped by you, and I'll try to be open about what I need.

I smile realising that since a long time, on the professional side of my life, I learnt to lead interaction with my customers along the same lines: "let's talk about what you need, not about what you expect of me".

Posted Fri Aug 14 12:13:16 2015 Tags:

Random quote

Be selfish when you ask, honest when you reply, and when others reply, take them seriously.

(me, late at night)

Posted Sun Jul 19 18:53:03 2015 Tags:

Billing an Italian public administration

Here's a simple guide for how I managed to bill one of my customers as is now mandated by law in Italy.

Create a new virtualbox machine

I would never do any of this to any system I would ever want to use for anything else, so it's virtual machine time.

  • I started virtualbox, created a new machine for Ubuntu 32bit, 8Gb disk, 4Gb RAM, and placed the .vdi image in an encrypted partition. The web services of Infocert's fattura-pa requires "Java (JRE) a 32bit di versione 1.6 o superiore".
  • I installed Ubuntu 12.04 on it: that is what dike declares to support.
  • I booted the VM, installed virtualbox-guest-utils, and de sure I also had virtualbox-guest-x11
  • I restarted the VM so that I could resize the virtualbox window and have Ubuntu resize itself as well. Now I could actually read popup error messages in full.
  • I changed the desktop background to something that gave me the idea that this is an untrusted machine where I need to be very careful of what I type. I went for bright red.

Install smart card software into it

  • apt-get install pcscd pcsc-tools opensc
  • In virtualbox, I went to Devices/USB devices and enabled the smart card reader in the virtual machine.
  • I ran pcsc_scan to see if it could see my smart card.
  • I ran Firefox, went to preferences, advanced, security devices, load. Module name is "CRS PKCS#11", module path is /usr/lib/opensc-pkcs11.so
  • I went to https://fattura-pa.infocamere.it/fpmi/service and I was able to log in. To log in, I had to type the PIN 4 times into popups that offered little explanations about what was going on, enjoying cold shivers because the smart card would lock itself at the 3rd failed attempt.
  • Congratulations to myself! I thought that all was set, but unfortunately, at this stage, I was not able to do anything else except log into the website.

Descent into darkness

Set up things for fattura-pa

  • I got the PDF with the setup instructions from here. Get it too, for a reference, a laugh, and in case you do not believe the instructions below.
  • I went to https://www.firma.infocert.it/installazione/certificato.php, and saved the two certificates.
  • Firefox, preferences, advanced, show certificates, I imported both CA certificates, trusted for everything, all my base are belong to them.
  • apt-get install icedtea-plugin
  • I went to https://fattura-pa.infocamere.it/fpmi/service and tried to sign. I could not: I got an error about invalid UTF8 for something or other in Firefox's stdandard error. Firefox froze and had to be killed.

Set up things for signing locally with dike

  • I removed icedtea so that I could use the site without firefox crashing.
  • I installed DiKe For Ubuntu 12.04 32bit
  • I ran dikeutil to see if it could talk to my smart card
  • When signing with the website, I chose the manual signing options and downloaded the zip file with the xml to be signed.
  • I got a zip file, unzipped it.
  • I loaded the xml into dike.
  • I signed it with dike.
  • I got this error message: "nessun certificato di firma presente sul dispositivo di firma" and then this error message: "Impossibile recuperare il certificato dal dispositivo di firma". No luck.

Set up things for signing locally with ArubaSign

  • I went to https://www.pec.it/Download.aspx
  • I downloaded ArubaSign for Linux 32 bit.
  • Oh! People say that it only works with Oracle's version of Java.
  • sudo add-apt-repository ppa:webupd8team/java
  • apt-get update
  • apt-get install oracle-java7-installer
  • During the installation process I had to agree to also sell my soul to Oracle.
  • tar axf ArubaSign*.tar*
  • cd ArubaSing-*/apps/dist
  • java -jar ArubaSign.jar
  • I let it download its own updates. Another time I did not. It does not seem to matter: I get asked that question every time I start it anyway.
  • I enjoyed the fancy brushed metal theme, and had an interesting time navigating an interface where every label on every icon or input field was truncated.
  • I downloaded https://www.pec.it/documenti/Manuale_ArubaSign2_firma%20Remota_V03_02_07_2012.pdf to get screenshots of that interface with all the labels intact
  • I signed the xml that I got from the website. I got told that I needed to really view carefully what I was signing, because the signature would be legally binding
  • I enjoyed carefully reading a legally binding, raw XML file.
  • I told it to go ahead, and there was now a .p7m file ready for me. I rejoiced, as now I might, just might actually get paid for my work.

Try fattura-pa again

Maybe fattura-pa would work with Oracle's Java plugin?

  • I went to https://fattura-pa.infocamere.it/fpmi/service
  • I got asked to verify java at www.java.com. I did it.
  • I told FireFox to enable java.
  • Suddenly, and while I was still in java.com's tab, I got prompted about allowing Infocert's applet to run: I allowed it to run.
  • I also got prompted several times, still while the current tab was not even Infocert's tab, about running components that could compromise the security of my system. I allowed and unblocked all of them.
  • I entered my PIN.
  • Congratulations! Now I have two ways of generating legally binding signatures with government issued smart cards!

Aftermath

I shut down that virtual machine and I'm making sure I never run anything important on it. Except, of course, generating legally binding signatures as required by the Italian government.

What could possibly go wrong?

Posted Thu Jul 2 23:48:36 2015 Tags:

debtags rewritten in python3

In my long quest towards closing #540218, I have uploaded a new libept to experimental. Then I tried to build debtags on a sid+experimental chroot and the result runs but has libc's free() print existential warnings about whatevers.

At a quick glance, there are now things around like a new libapt, gcc 5 with ABI changes, and who knows what else. I figured how much time it'd take me to debug something like that, and I've used that time to rewrite debtags in python3. It took 8 hours, 5 of pleasant programming and the usual tax of another 3 of utter frustration packaging the results. I guess I gained over the risk of spending an unspecified amount of hours of just pure frustration.

So from now on debtags is going to be a pure python3 package, with dependencies on only python3-apt and python3-debian. 700 lines of python instead of several C++ files built on 4 layers of libraries. Hopefully, this is the last of the big headaches I get from hacking on this package. Also, one less package using libept.

Posted Sun Jun 21 18:04:39 2015 Tags:
05

Internet references saved for May 2015

Instead of keeping substantial tabs open until I have read all of them, or losing them in the jungle of browser bookmarks, I have written a script that collects them into a file per month, and turns them into markdown files for my blog. This way I sort of know where to find them, and if I do not, some internet search might. And if I wish, I can even choose to share it.

  • The programming talent myth [LWN.net]

    Jacob Kaplan-Moss is known for his work on Django but, as he would describe in his keynote, many think he had more to do with its creation than he actually did. While his talk ranged quite a bit, the theme covered something that software development organizations—and open source projects—may be grappling with: a myth about developer performance and how it impacts the industry. It was a thought-provoking talk that was frequently punctuated by applause; these are the kinds of issues that the Python community tries to confront head on, so the talk was aimed well.

  • Weblog for dkg - Preferred Packaging Practices

    git-buildpackage-based packaging practices from dkg

  • Matt Palmer: The Vicious Circle of Documentation

    Why are there so many more undocumented systems than documented ones out there, and how can we cause more well-documented systems to exist? The answer isn’t “people are lazy”, and the solution is simple – though not easy.

  • German Courses | DW.DE

    Free German courses

  • Jono Bacon, Dealing With Disrespect

    This book is about helping us to focus on good people creating good things, to preserve that spirit of sharing, and to protect against those whose primary contribution is obstruction and disrespect

  • Stack Overflow: unix - What are the dark corners of Vim your mom never told you about?

    Collection of vim tips that people actually use

  • OFTC: Automatically Identifying Using SSL + CertFP

    This page describes how to use SSL with a certificate fingerprint to automatically identify your registered nickname with NickServ on connect. You must have an IRC client that supports SSL with a client certificate.

  • François Marier: Usual Debian Server Setup

    I manage a few servers for myself, friends and family as well as for the Libravatar project. Here is how I customize recent releases of Debian on those servers.

  • The public SIP and ENUM service | sip5060.net

    SIP service, also providing test call services for SIP clients

  • How to name things: the hardest problem in programming

    Developers can get better at their craft by learning from the great writers who mastered theirs. Writing software isn’t the same as writing a novel, but there are parallels. Besides, advice from writers is better because writers have been struggling with their craft for many centuries, not just a few decades. It’s better-written as well. This talk shares great writers’ best advice for coders: Stephen King on refactoring, Anne Rice on development hardware, Hemingway on modelling with personas, and Neil Gaiman on everything.

download as mailbox

Posted Mon Jun 1 12:29:23 2015 Tags:

Love thy neighbor as thyself

‘Love thy neighbor as thyself’, words which astoundingly occur already in the Old Testament.

One can love one’s neighbor less than one loves oneself; one is then the egoist, the racketeer, the capitalist, the bourgeois. and although one may accumulate money and power one does not of necessity have a joyful heart, and the best and most attractive pleasures of the soul are blocked.

Or one can love one’s neighbor more than oneself—then one is a poor devil, full of inferiority complexes, with a longing to love everything and still full of hate and torment towards oneself, living in a hell of which one lays the fire every day anew.

But the equilibrium of love, the capacity to love without being indebted to anyone, is the love of oneself which is not taken away from any other, this love of one’s neighbor which does no harm to the self.

(From Herman Hesse, "My Belief")

I always have a hard time finding this quote on the Internet. Let's fix that.

Posted Wed May 20 11:35:15 2015 Tags:

Work around Google evil .ics feeds

I've happily been using 2015/akonadi-install for my calendars, and yesterday I added an .ics feed export from Google, as a URL file source. It is a link in the form: https://www.google.com/calendar/ical/person%40gmail.com/private-12341234123412341234123412341234/basic.ics

After doing that, I noticed that the fan in my laptop was on more often than usual, and I noticed that akonadi-server and postgres were running very often, and doing quite a lot of processing.

The evil

I investigated and realised that Google seems to be doing everything they can to make their ical feeds hard to sync against efficiently. This is the list of what I have observed Gmail doing to an unchanged ical feed:

  • Date: headers in HTTP replies are always now
  • If-Modified-Since: is not supported
  • DTSTAMP of each element is always now
  • VTIMEZONE entries appear in random order
  • ORGANIZER CN entries randomly change between full name and plus.google.com user ID
  • ATTENDEE entries randomly change between having a CN or not having it
  • TRIGGER entries change spontaneously
  • CREATED entries change spontaneously

This causes akonadi to download and reprocess the entire ical feed at every single poll, and I can't blame akonadi for doing it. In fact, Google is saying that there is a feed with several years worth of daily appointments that all keep being changed all the time.

The work-around

As a work-around, I have configured the akonadi source to point at a local file on disk, and I have written a script to update the file only if the .ics feed has actually changed.

Have a look at the script: I consider it far from trivial, since it needs to do a partial parsing of the .ics feed to throw away all the nondeterminism that Google pollutes it with.

The setup

The script needs to be run periodically, and I used it as an opportunity to try systemd user timers:

    $ cat ~/.config/systemd/user/update-ical-feeds.timer
    [Unit]
    Description=Updates ical feeds every hour
    # Only run when on AC power
    ConditionACPower=yes

    [Timer]
    # Run every hour
    OnActiveSec=1h
    # Run a minute after boot
    OnBootSec=1m
    Unit=update-ical-feeds.service

    $ cat ~/.config/systemd/user/update-ical-feeds.service
    [Unit]
    Description=Update ICal feeds

    [Service]
    # Use oneshot to prevent two updates being run in case the previous one
    # runs for more time than the timer interval
    Type=oneshot
    ExecStart=/home/enrico/tmp/calendars/update

    $ systemctl --user start update-ical-feeds.timer
    $ systemctl --user list-timers
    NEXT                         LEFT       LAST                         PASSED UNIT                    ACTIVATES
    Wed 2015-03-25 22:19:54 CET  59min left Wed 2015-03-25 21:19:54 CET  2s ago update-ical-feeds.timer update-ical-feeds.service

    1 timers listed.
    Pass --all to see loaded but inactive timers, too.

To reload the configuration after editing: systemctl --user daemon-reload.

Further investigation

I wonder if ConditionACPower needs to be in the .timer or in the .service, since there is a [Unit] section is in both. Update: I have been told it can be in the .timer.

I also wonder if there is a way to have the timer trigger only when online. There is a network-online.target and I do not know if it is applicable. I also do not know how to ask systemd if all the preconditions are currently met for a .service/.timer to run.

Finally, I especially wonder if it is worth hoping that Google will ever make their .ics feeds play nicely with calendar clients.

Posted Wed Mar 25 21:50:21 2015 Tags:

Screen-dependent window geometry

I have an external monitor for my laptop in my work desk at home, and when I work I keep a few windows like IRC on my laptop screen, and everything else on the external monitor. Then maybe I transfer on the sofa to watch a movie or in the kitchen to cook, and I unplug from the external monitor to bring the laptop with me. Then maybe I go back to the external monitor to resume working.

The result of this (with openbox) is that when I disconnect the external monitor all the windows on my external monitor get moved to the right edge of the laptop monitor, and when I reconnect the external monitor I need to rearrange them all again.

I would like to implement something that does the following:

  1. it keeps a dictionary mapping screen geometry to window geometries
  2. every time a window geometry and virtual desktop number changes, it gets recorded in the hash for the current screen geometry
  3. every time the screen geometry changes, for each window, if there was a saved window geometry + wirtual desktop number for it for the new screen geometry, it gets restored.

Questions:

  1. Is anything like this already implemented? Where?
  2. If not, what would be a convenient way to implement it myself, ideally in a wmctrl-like way that does not depend on a specific WM?

Note: I am not interested in switching to a different WM unless it is openbox with this feature implemented in it.

Posted Mon Mar 16 21:29:36 2015 Tags:
Posted Sat Jun 6 00:57:39 2009
dcg

Debian Community Guidelines.

Reorganization of the DCG

I've recently received a substantial amount of feedback about the Debian Community Guidelines and went into some reorganization of it.

The previous general section still stands as the Main Guidelines: those are the substantial few things to always keep in mind.

What previously was the long list of checklists is now split in two: the Debian-specific Guidelines, which should be a shorter lists of non-obvious suggestions for people who already have experience with online life, and the General Guidelines, with the fuller checklists with useful suggestions for everyone.

I still haven't gone through the selection and reorganization of the Debian-specific and General part, so at the moment they look fairly similar and most things overlap. But the good news is that I finally found a structure that I like, and that can allow more experienced people to make use of the guidelines without getting bored with simpler things like "google before asking a question".

This division also suggests a little workflow: new suggestions can be added to the Debian-specific part, and then later moved to the general part when they become obvious for everyone.

I'm happy. This layout seems to be good in getting me unstuck with how to think of the DCG. More will come of course, as I'll prepare my DCG talk for Debconf6.

Posted Sat Jun 6 00:57:39 2009 Tags:

Debconf6 talks material now online

I've finally put online slides and notes for my debconf6 talks:

Many people had asked me the notes for the "Advanced ways of wasting time" talk: they're finally online, translated and with the links pointing to English Wikipedia pages. Sorry it took me so long.

Posted Sat Jun 6 00:57:39 2009 Tags:

DCG mentioned on Linux.com

A friend of mine pointed me to the Debian Community Guidelines being mentioned in a linux.com article.

"[Mako] hopes that Garrett's resignation will give the Debian community an added impetus to adapt its own code of conduct, like the one proposed by Enrico Zini."

I'm very happy to see the DCG geting mentioned, although I don't think that it makes sense to 'adopt' such a document.

what I'd like for it is to be mentioned as a suggested read, and linked from here and there. So today it happened, and I'm happy :)

Posted Sat Jun 6 00:57:39 2009 Tags:

Converging to a solution

Sustain a discussion towards solving a problem is sometimes more important than solving the problem.

I can't decide if this is trivial or counterintuitive. Anyway it's been quite enlightening when it came out. I once took this note:

I found that with my projects, when someone posted a mail about a problem I would work maybe some days to find a solution, and just post the solution at the end.

However, now I realised it's more costructive to have the problem-solving process itself happen online. This way, instead of keeping people waiting in silence for a few days they can get quicker feedback and extra informations, and they also have a chance to participate to solving the problem before I manage to.

For example, when I have to interrupt to go home or sleep, someone else can pick it up and do another step.

Plus, the entire problem-solving process remains documented, which will provide more written information for future readers.

This note was from a few months ago; however, I still fail to do it. Bad habits are sometimes hard to change. Please kick me about it :)

Posted Sat Jun 6 00:57:39 2009 Tags:
Posted Sat Jun 6 00:57:39 2009