Crypto Crap in Python

I’m looking into doing a little cryptographic stuff in python. Nothing fancy, just some standard stuff. Not for the first time I’m bumping into this brick wall of “batteries included”, the notion that the python library comes with a lot of stuff that should be good enough for whatever you need to do. Only problem is that it doesn’t. XML parsing stinks in Python; http IO stinks (need lots of third party stuff to make that usable); no UTF-8 by default; etc.

Out of the box python is bloody useless unless you want to do some very simplistic stuff. So basically my problem is very simple: I need to be able to sign stuff and verify signatures in a way that is compatible with how stuff like this stuff is commonly done on the internet ™. I.e. you’d expect some pretty mature, well tested libraries to be around for whatever programming language you’d like to use. I know exactly where to go to get this stuff for Java, for example.

So we’re looking at some very basic capability to do stuff with algorithms like RSA, SHA1, MD5 etc. Batteries not included with python at all so I Google a bit to find out what people commonly use for this in python and stumble upon what seems to be the most popular library pycrypto. It seems to have all the algorithms, great! Only one minor detail that has had me crawl all over Google for the entire afternoon:

Public keys usually come as base64 encoded thingies: how the hell do I get them in and out of the functions/classes and what not provided by pycrypto. Batteries not included. After a long search, I find this nice post.

Basically it’s telling me that various people have bothered to provide nice libraries with relevant code for python but somehow all of them have neglected to provide this very basic functionality that you will need 100% guaranteed. That just sucks. In the hypothetical case that you’d actually want to use this stuff to do hypothetically useful things like verifying a signature attached to some http request you will basically find yourself reverse engineering this poorly documented library and figuring out how to get from a base 64 encoded RSA key to a properly configured RSA class instance and back again. I had lots of fun (not) reading about the details of RSA, x.509, etc.

Eventually I found some sample code here that seems to half do what I need. But I’d just prefer to be able to reuse something that is hassle free instead of copy pasting somebody else’s code and debugging it until it works as expected and basically reinventing the wheel by making what would amount to Jilles private little python crypto library. I have better things to do.

Web application scalability

It seems infoq picked up some stuff from a comment I left on the serverside about one of my pet topics (Server side Java).

The infoq article also mentions that I work at Nokia. I indeed work for Nokia Research Center and it’s a great place to work. Only they do require me to point out that when making such comments I’m not actually representing them.

The discussion is pretty interesting and I’ve recently also ventured into using other things than Java (mainly python lately with the Django framework). So far I dearly miss development tooling which ranges from non existent to immature crap for most languages that are not Java. Invariably the best IDEs for these languages are actually built in Java. For example, I’m using the eclipse pydev extension for python development. It’s better than nothing but it still sucks compared to how I develop Java in the same IDE. Specifically: no quickfixes; only a handful of refactorings, no inline documentation, barely working autocompletion, etc make life hell. I forgot what it is like to actually have to type whole lines of code.

I understand the development situation is hardly better for other scripting languages. There’s some progress on the ruby front since Sun started pushing things on that side but none of this stuff is actually production quality. Basically the state of the art in programming environments is currently focussed primarily on statically compiled OO languages like Java or C#. Using something else can be attractive from for example language feature point of view but the price you pay is crappy tooling.

Python as a language is quite OK although it is a bit out of date with things like non utf-8 strings and a few other things that my fellow country man Guido van Rossum is planning to fix in python 3000. Not having explicit typing takes some getting used to and also means my workload is higher because I constantly have to use Google to look up stuff that eclipse would just tell me (e.g. what methods and properties can I use on this HttpResp object I’m getting from Django; what’s the name of the exception I’m supposed to be catching here, etc). In my view that’s not progress and leads to sloppy coding practices where people don’t bother dealing with fault situations unless they have to (which long term in a large scale server environment is pretty much always).

Kdiff3 to the rescue

I was struggling this evening with the default merge tool that ships with tortoise svn. It’s not bad and quite user friendly. However, I ran into trouble with it when trying to review changes in a latex file (don’t ask, I still hate the concept of debugging and compiling stuff I would normally type in word). The problem was that it doesn’t support word wrapping and that the latex file in question used one line per paragraph (works great in combination with an editor that does soft word wrapping like e.g. Jedit).

A little googling revealed that the problem had been discussed on the tortoise svn mailing list and dismissed by one of the developers (for reasons of complexity). Figuring that surely somebody must have scratched this itch I looked on and struck gold in the form of this blogpost:KDiff3 – a new favorite about KDiff3.

The name suggests that this is a linux tool. Luckily it seems there is a windows port as well so no problem here. I installed it and noticed that by default it replaces the diff editor in tortoisesvn (good in this case but I would have liked the opportunity to say no here). Anyway, problem solved :-). A new favorite indeed.

Update:

Nice little kdiff3 moment. I did an update from svn and it reported a python file was in conflicted state. So I dutifully right clicked and selected edit conflicts. This launched kdiff which reported: 4 conflicts found; 4 conflicts automatically resolved. It than opens into a four pane view (mine, base, theirs and merged) allowing you to easily see what the merged result looks like and what the conflicts were. OMFG! where were you all this time kdiff3!! Damn that is useful. The resolutions look good too. I remember using tortoise svn doing merges on very large source base in my previous job and this is exactly what made them suck so much.

Moving time

I’m not much into life logging, I prefer to stick to technology :-). But under the circumstances, I’ll make an exception.

As you may know, I recently got a job as a research engineer at the Nokia Research Center in Helsinki. That means I am going to leave Nijmegen and the Netherlands soon. Soon as in next week. It’s been a bit more than five years since I moved back to NL from Sweden and now I’ll move to Finland. Just like the previous time indefinately, meaning that I’ll move back when I feel like moving back again.

I had my last day at GX last wednesday. Currently I am packing some stuff and tonight there will be drinks at Maxim (a cafe in Nijmegen) for my friends and colleagues. Tomorrow, after the hangover becomes tolerable, I’ll visit my parents in Breda and say hello to my sister. Then Monday the moving people will pack & pick up my stuff and hopefully deliver it to the apartment I will hopefully find real soon after I get to Helsinki, which is on Tuesday.

On a side note, ever since I got the strange idea of moving to Finland I’ve frequently been humming/whistling/etc. the Finland song:

Finland, Finland, Finland,
The country where I want to be,
Pony trekking or camping,
Or just watching TV.
Finland, Finland, Finland.
It's the country for me.

You're so near to Russia,
So far from Japan,
Quite a long way from Cairo,
Lots of miles from Vietnam.

Finland, Finland, Finland,
The country where I want to be,
Eating breakfast or dinner,
Or snack lunch in the hall.
Finland, Finland, Finland.
Finland has it all.

You're so sadly neglected
And often ignored,
A poor second to Belgium,
When going abroad.

Finland, Finland, Finland,
The country where I quite want to be,
Your mountains so lofty,
Your treetops so tall.
Finland, Finland, Finland.
Finland has it all.

All together, Finland fans!
Finland, Finland, Finland,
The country where I quite want to be,
Your mountains so lofty,
Your treetops so tall.
Finland, Finland, Finland.
Finland has it all.

Finland has it all.

(Monthy Python). Great song.

For those I won’t see anymore: it was nice knowing you and maybe we’ll meet again.