My rule to see if a framework is worth of attention

I came up with a little rule:

In order to be worth of any attention, a framework must be stable enough that I can charge money to train people to use it.

This probably applies to other kinds of software stacks, libraries, development environments and, well, to most software applications.

In the context of python web frameworks, this means that:

• If it changes API all the time it is not worth of attention, because my customers won't get value for their money, as they'd continuously need retraining and rewriting their software.
• If I see lots of DeprecationWarnings it is not worth of attention, because my customers will see them and blame me for teaching them deprecated stuff.
• If fixes for bugs affecting the stable version are only distributed "in a recent git" or "in the next development version", and they are not backported into a new bugfix-only stable release, then it is not worth of attention, because:
  • my customers' business is to develop their own products based on the framework.
  • My customers' business is not to be maintaning in-house stable updates of the framework. Although if the framework's community is nice enough they might end up giving a hand.
• If it requires virtualenv or can only be obtained through easy_install it is not worth of attention, because:
  • my customers are not interested in maintaning custom deployment environments over time.
  • My customers are not interested in tracking each and every single library's upstream development to keep their production system free of bugs.
  • My customers are used to getting software through a proper distribution which also takes care of security updates.
  • I am paid to teach them how to use a framework, not a custom python-only package management system.
  • In my experience, if distributions have trouble keeping packages up to date, upstream is doing something fundamentally wrong.

In light of this rule, I regret to notice that I see very few python web frameworks worth of any attention.

On python stable APIs

There is a theory which states that if ever anyone discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable.

There is another theory which states that this has already happened.

In Debian testing:

/usr/lib/python2.6/dist-packages/sqlalchemy/types.py:547: SADeprecationWarning: The Binary type has been renamed to LargeBinary.

In Debian Lenny:

ImportError: cannot import name LargeBinary

I was starting to think that SQLAlchemy wasn't too bad, since I've been using it for 6 months and I haven't seen its API change yet.

But there it is, a beautiful reminder that SQLAlchemy, too, is part of the marvelously autistic Python ecosystem.

Released cfget 0.16

I have released version 0.16 of cfget.

It is just a little bugfix release as I found a bug in the new expression parser, and while I was at it I simplified its code quite a bit.

On python, frameworks and TOOWTDI

The Python world is ridden with frameworks, microframeworks, metaframeworks and their likes. They are often very clever things, but more often than not they are a tool of despair.

A very peculiar thing about Python web frameworkish things is that there are so many of them. There's cherrypy (in its various API redesigns), fapws, gunicorn, bottle and flask, paste, werkzeug and flup, tornado, pylons, turbogears 1 and 2, django, repoze who, what and whatnot, all the myriad of rendering engines and buffet as a metathing on top of them, diesel, twisted, and I apologise if I don't spend my day listing and hyperlinking them all, I hope I made my point.

Frameworks are supposed to standardise some aspects of programming; the nice thing about standards is that there are so many of them to choose from, and they all suck, so I'll make my own.

But wasn't Python supposed to be the world of TIOOWTDI?

Ok, everbody knows it isn't. Just in the standard library there are 2 implementations of pickle and 2 urllibs. But people like the TIOOWTDI idea.

I believe the reason people like the TIOOWTDI idea is because it creates a framework. It standardises some aspects of programming, and defines building blocks that guarantee that people doing similar jobs will be using similar sets of components.

Let's take for example the datetime module in the standard library. It is an embarassing example of a badly designed module, so embarassing that the standard library documentation continuously fails to document its fundamental design flaws and common work-arounds hoping that noone notices them, but as a consequence each poor soul starting to use it for nontrivial things has to google for hours in despair to rediscover in how many ways it's broken.

But still, datetime works as a structure to hold those values that make a date, time, or full UTC timestamp. For that job it's become the standard, and as such it's an important component of the Python TIOOWTDI framework: one can use it to exchange datetimes among different libraries: for example ORMs are using it instead of rolling their own, which makes database programming so much easier when date/time is involved.

Even if the implementation is far from perfect, once we apply TIOOWTDI to dates and timestamps, python code from different authors can exchange dates without worries. This is much better than having 3 different superior datetime libraries and having to convert date objects from one to another when passing values from a web form to an ORM.

There is an often overlooked Python framework. The Pyton framework. It's called TIOOWTDI.

All the micro-mini-midi-maxi-meta-frameworks that people scatter around, are, or should be, just experiments, proofs of concept, competing ideas waiting to be distilled in The Only One Way, bringing the Python experience one step forward.

What is unfortunate is that this last distillation thing happens so rarely that people get used to the idea of having to use proof of concept code to get things done.

Update:

this post apparently wasn't very clear, so here is some clarification:

• that python, plus the idea of TIOOWTDI, in fact generate a framework, the framework;
• that framework currently includes very little web stuff;
• the web stuff is currently shipped in countless prototypes and proof of concept things;
• something from all the prototypes people develop ought to be brought into the TIOOWTDI "framework";
• otherwise people get used (or worse, forced) to use proof of concept throwaway code to get their job done.

Released cfget 0.15

I have released version 0.15 of cfget.

cfget is a tool to extract values from ini-style config files. A trivial thing, really.

It is also simple to install: it is a single python executable and it has no dependencies besides the python standard library.

It is trivial and simple, but because of the complex requirements (and sponsorship) of ISAC - CNR it has recently accumulated quite a set of features, and it manages to get a remarkable lot of things done.

There are several news since 0.8, worth a rather major announcement:

Added --dump=pickle

Now all the contents of the configuration files, plus all the contents generated by cfget plugins, can be dumped in pickle format.

This provides a quick and dirty way to load all cfget-generated data into a python dict:

data = pickle.loads(
    subprocess.Popen(
        ["cfget", "--dump=pickle"],
        stdout=subprocess.PIPE).communicate()[0])

It sounds like a rather complicated way to read a configuration file, but if you have various plugins that compute nontrivial derived configuration values, that quick&dirty hack could be quite useful.

Curly brace expansion

Suppose that you have a config file like this:

[general]
mode = show

[show]
command = display

[edit]
command = gimp

And you want to get the command from the section indicated by general/mode.

Now you can, very easily: cfget '{general/mode}/command'

When it notices curly braces, cfget will literally replace them with the result of querying their contents, then parse the expression again.

Simple expression support

ISAC - CNR are using cfget to configure the run of a rather complicated physical model, and use plugins to derive all sorts of values form the base configuration.

This works, but there are times when adding a function to a plugin sounds like overkill: for example, sometimes one needs foo/bar + 1, or just the hour of a timestamp.

For those simple cases, I've added support for simple expressions:

• operators: +, -, *, /, **
• grouping with parentesis
• function calls (int(), round()), with the possibility to define new functions via plugins.

So to compute a middle point one can now do this:

cfget "round((pos/start + pos/end) / 2)"

It needs a space around arithmetic operators to avoid conflicts with characters used to refer to configuration values, but with the space the expressions look nicer, so the result is that it generally does the right thing.

Just as a scary thought, curly braces work with and in expressions:

cfget "values/val{round((pos/end - pos/start) / 2)} + 1"

Although if someone ends up having a hairy thing like that, it is worth considering to replace it with a dynamic value computed using a plugin.

A golden rule in Debian things

If there's something that doesn't go the way you think, write:

I don't understand, please explain me

instead of:

You don't understand, I'll explain you

In the first case, you'll likely get an answer that starts with "I'm sorry". In the second case, you'll likely FAIL.

On software modularity

Software modularity is good. You take many modular, reusable components and you join them together to create a bigger software that is more than the sum of its parts:

But what happens if you try to put together all sorts of modular parts, each independently designed by a smart person who believes that their way of doing things is the best in the world?

It's not exactly easy to save the world in that.

This post has been brought to you after a lot of paid work on Python, WSGI and all those sorts of mini/midi/maxi/meta/macro frameworks people built with them, that always look really, really cool until you actually start to use them for getting a serious job done.

Released nodm 0.7

I have released version 0.7 of nodm.

It only fixes one silly typo in autotools, which made it fail to build on Fedora.

fuss-launcher: an application launcher built on apt-xapian-index

Long ago I blogged about using apt-xapian-index to write an application launcher.

Now I just added a couple of new apt-xapian-index plugins that look like they have been made just for that.

In fact, they have indeed been made just for that.

After my blog post in 2008, people from Truelite and the FUSS project took up the challenge and wrote a launcher applet around my example engine.

The prototype has been quite successful in FUSS, and as a consequence I've been asked (and paid) to bring in some improvements.

The result, that I have just uploaded to NEW, is a package called fuss-launcher:

* New upstream release
   - Use newer apt-xapian-index: removed need of local index
   - Dragging a file in the launcher shows the applications that can open it
   - Remembers the applications launched more frequently
   - Allow to set a list of favourite applications

To get it:

• apt-get install fuss-launcher (after it passed NEW);
• or git clone http://git.fuss.bz.it/git/launcher.git/ and apt-get install python-gtk2 python-xapian python-xdg apt-xapian-index app-install-data

It requires apt-xapian-index >= 0.35.

To try it:

1. Make sure your index is up to date, especially if you just installed app-install-data: just run update-apt-xapian-index as root.
2. Run fuss-launcher.
3. Click on the new tray icon to open the launcher dialog.
4. Type some keywords and see the list of matching applications come to life as you type.

It's worth mentioning again that all this work was sponsored by Truelite and the Fuss project, which rocks.

Some screenshots:

When you open the launcher, by default it shows the most frequently started applicationss and the favourite applications:

When you type some keywords, you get results as you type, and context-sensitive completion:

When you drag a file on the launcher you only see the applications that can open that file:

New apt-xapian-index plugins

Besides a fair bit of refactoring and cleanup, I've recently added two new plugins to apt-xapian-index:

app-install

If app-install-data is installed, information about .desktop files will now enter the index.

This allows, for example, to limit query results to only those packages that contain .desktop files, which is quite useful, for example for building desktop-oriented package managers.

aliases

It reads term->aliases mapping from files in /etc/apt-xapian-index/aliases/ or /usr/share/apt-xapian-index/aliases/, and feeds them as synonyms in the index.

apt-xapian-index ships an example alias file, to give people who know the wrong software names a chance to find the right ones:

# Aliases expanding names of popular applications

excel       XToffice::spreadsheet
powerpoint  XToffice::presentation
photoshop   XTworks-with::image:raster
coreldraw   XTworks-with::image:vector
autocad     XTworks-with::3dmodel

Notice how it is possible to use index terms that happen to be Debtags tags as synonyms, which yields better results, language independence and extra coolness.