My general solution to this problem has been to simply not do a review of any Nokia product unless I actually like it and won’t have to lie to you. So yes, I’m biased and definitely not objective but at the same time I don’t work for marketing, and definitely would feel bad about propagating lies.

I had no problems doing an N900 review a while back or reviewing the N8. So, now it is the Lumia 800′s turn. The good news is, I like this phone. In fact, like it so much I’d recommend this phone and its little brother the 710 to anyone, including close friends and family whom I definitely would not want to burden with a bad product. To be clear, I don’t take this type of recommendation lightly because if I’m wrong, I basically have to deal with the consequences of people complaining about embarrassing short comings, etc.

Much has been said in the press already about the Lumia line of phones. The general mood seems to be that reviewers like the phone but with some reservations. Some of the criticism that is repeated a lot by different reviewers are mostly non issues in my view. I’ll try to cover the essentials here and try to counter some of the criticism or at least put it in perspective:

1. The windows phone market place is far smaller than the iphone and android stores. This is a big one, almost every review repeats it. It’s the single most important argument against windows phone in general and the Lumia in particular. And it is true, the long tail of useless fart applications, rss feeds for obscure websites, pointless applications, etc. is a lot shorter in the market place. But guess what, almost nobody installs those anyway. What matters are the handful of important applications that are actually popular. And while there are some things missing, most of the important functionality seems to be covered in the market place and there seem to be lots of developers filling in the gaps quickly. Additionally, the phone actually has a lot of well integrated support for social networks that removes a lot of the need for third party apps. So, even though facebook and twitter apps exist, I actually don’t really use them because the people hub handles things just fine for me and I can upload photos straight from the photo application. Having apps is nice. Not needing them because the phone does it out of the box is better. Finally, the mobile browser is pretty decent and allows you to run a lot of mobile web sites just fine: why have an app if a bookmark can do the job? Also, you can pin websites to the home screen. I’ve got Google News and Google Reader for touch screen phones there (google.com/reader/i). The boundary between app and web app is pretty blurry on this phone. In any case, I have about 50 apps and games installed on this phone, several of which I use regularly. See below for a list of my favorites.
2. Battery life is not great. This is partially due to the small capacity of the battery but also due to a range of software issues that are being patched. Just today I installed an update that seems to be extending battery life for me quite a bit. Normally my phone would need charging somewhere during the night and would be well below 30-40% in the evening (i.e. now). Current battery level is 70%. Even before this patch, I’ve rarely run out of power. I charge at night normally and this provides me enough juice to last me a day of admittedly generally not so heavy usage.
3. Multi tasking is limited. This is true, but it helps keeping the phone alive. The important things do work in the background though: I can play music or online radio in the background; email and social networks update themselves and if you are like the constant distraction of being IMed from various IM networks, that works fine as well. Apple does this as well: full blown multi tasking is actually not a great idea on a mobile phone with limited CPU and RAM. Android has unrestricted multi tasking and a whole lot of battery and performance issues that I don’t think are worth the trouble.
4. The hardware is limited. I would say, windows phone performs great on modest hardware. The phone has a single core, 1.4 Ghz CPU with 512MB of RAM. Especially the ram size is very modest. The CPU is actually plenty fast and since there is little need for multi tasking, why bother with a second CPU core? Technicalities aside, this phone is lag free, blazing fast and responsive, all the time. I’ve not seen this phone freeze or lag in the three months I’ve been using it. I think few Android phone owners can claim that and even Apple’s iphone has had its fair share of issues.
5. The lid on the top that covers the usb port is fragile. This is true but the way it is designed this is only an issue if it is open. Most of the time it will be closed when you are handling the phone and while it opens easily it never does so accidentally. Which is the reason that most Lumia 800′s I’ve seen at work (and there is a pretty large number of them) are undamaged in that respect. I’ve seen exactly one phone where it broke off so far.

So, now on to the rest of this review which is mainly about outlining why I like this phone so much.

First of all, world + dog seems to agree this is a really nicely designed phone. No argument there. It is a nicely sized phone that fits in my pocket and doesn’t make me look like I’m happy to see everyone. Also it gets along with the key ring that I keep in that pocket just fine. So far I have not managed to scratch or damage this phone in any significant way, which is impressive. You should see some of my other phones, not pretty.

The screen is great. It’s a nice Oled screen with very good contrast and blacks that are virtually indistinguishable from the phone cover, which is also black. Nice crisp colors too and usable in the desert and on the beach as I found out in Australia recently. It’s great for viewing photos and watching movies.

Speaking of photos and movies, shooting those delivers very nice results with the 5mp camera. With 720p you get a pretty nice quality and the sound is pretty decent too. This is a very decent camera phone.

But the real reason I like this phone is the software. Prior to windows phone I did not believe that windows on a phone would be any good. I stand corrected. It works really well and Microsoft did not just level the playing field here, they raised the bar for everyone else. It’s that good: expect to see a lot of imitation from other vendors in the next year or so.

The metro UI is very nice to use and basically obsoletes the notion of having screens full of static icons. Instead, metro is centered around tiles. Tiles basically act like a combination of widgets and icons. Tapping one opens an application. Tiles update themselves with useful information (like the weather, a photo somebody posted, etc.) so they are useful even if you don’t tap them. For example I have separate tiles for messaging, calendar, and various mail accounts that allow me to see whether I need to tap them with a quick glance at the screen. All tiles are the same size and this makes navigating them easy. Instead of having multiple screens, there is just one endless screen that you can flick up and down as you please. You can long tap to drag and drop tiles to rearrange things and I basically have all the important ones listed at the top and the ones I care less about at the bottom where they are still only a quick flick with my finger away. Clearly a lot of consideration and design went into this UI and it works really well for me.

What is especially nice is the level of integration between applications. Few reviews highlight this enough but the people hub is a really nice piece of software that completely revolutionizes what used to be a boring phone book full of faceless people and phone numbers. I can tweet from it, I can read other people’s updates, view photos from several networks, see a contact details that are enriched with stuff from several profiles on twitter, linkedin, facebook, gmail, outlook, hotmail, and more. Also, it’s smart enough to identify duplicates and suggesting contacts that could be merged. Few other phones get this right. Additionally, you can group people and have tiles for those groups both for quick access but also as a way of filtering their status updates. For example, I have a friends and a family tile on my home screen, in addition to the main people hub tile.

There is a special me tile that allows you to review your own contributions and do status updates. The same accounts I have in the people hub are also available for photo sharing in the photo application. You can share a photo right after taking it or from the album.

The other thing that surprised me in a positive way is the browser, which is a mobile version of internet explorer 9. I’ve been a long time firefox user and have basically hated internet explorer on the desktop with a passion for years, version 9 included. However, on mobile it works surprisingly well. Version 9 brought support for a meaningful enough subset of html 5 that most sites are usable. Sites load relatively quick compared to other browsers and zooming and navigating pages works fine as well. Compared to mobile versions of firefox on the N900 and the symbian webkit based browser as well as opera for symbian, this is a really nice browser that offers both the performance and usability needed to be useful.

Some apps that I like and use include:

• Nokia Transport, Nokia Maps, Nokia Drive. All made by my colleagues here in Berlin and powered by my own work on the back end software. But still, these are fine applications that are useful now in their first version for WP 7 and are going to get even better very soon as we deliver more of the feature work.
• Nokia Creative Studio. A very decent photo editing application that was released recently. Pretty neat and the live view is a great way to take photos.
• Kindle for windows phone is great. I also have a normal kindle and the two sync nicely via whispersync.
• I have two weather applications: AccuWeather and a Microsoft “Weather” application. Both are location aware and have live tiles and do a decent job of telling you the weather now and in the near future.
• The youtube mobile website works really well. I can basically view my subscriptions and movies play back very nicely full screen on the fantastic screen.
• Tunein Radio offers an extensive selection of most relevant radio channels world wide.
• IMDB has a nice application that is location aware and updates with news and reviews for movies that are playing nearby as well as the schedules for those movies.
• Foursquare has a pretty decent application with live tiles.
• Connectivity short cuts gives you tiles for quick access to bluetooth, wifi, and cellular, and airplane settings.

In recent years I’ve observed people making outrageous claims about the supposed benefits of things like scrum or kanban while in fact they were actively making things less agile. I’ve experienced first hand some botched roll outs of scrum. And the software engineering researcher inside of me has been itching for a while to write up my perspective on the whole thing. So, here it goes.

The word agile itself is a bit of a misnomer in the context of software development. It intends to communicate a intent to deliver software fast and efficiently. This requires that people do things in a systematic and appropriate way. That’s where the madness begins because most people will equate this to a dogmatic application of whatever software development methodology they favor. Dogmatic is in my book a synonym for stupid: you switch off your brain and blindly do whatever the dogma dictates. This is what is going on with agile software development. People go to conferences, hear opinions and sales pitches and accept them as fact. Then they go back to work and do a lot of damage.

If you read up on agile methodologies, you will find that their lead proponents (e.g. Kent Beck, Martin Fowler, Alistair Cockburn, etc.) are generally a bit more pragmatic and rather anti dogmatic than their followers and fan boys. The same goes for the proponents of scrum, including Ken Schwaber. I have great respect for these people and the fine work they have done moving the field forward.

These people are successful veteran software developers or project managers with a lot of successful projects behind them. That in my view is the main problem here: you can’t substitute experience with process and that is exactly what agile followers end up doing. Take away the experienced people and you have a lot of people executing the rituals of scrum, kanban, or whatever without really understanding what they are doing. The assumption that that will work at all is naive at best and actually rather delusional. In this article, I’ll focus mostly on scrum because that is the dominant agile methodology currently and I’ve had some first hand experience with it in my current job.

A good software organization will have key people that keep things together technically and organizationally and a larger group of people with various skills and qualities that do stuff that they’ve been told to do but are not necessarily capable of doing this unsupervised. With a good mix of those people working together, you can achieve great things fast. A bad scrum implementation will empower the latter group while taking power away from the former group. I say bad scrum implementation because in theory there are good implementations as well. It’s just I haven’t seen too many of those (i.e. none) and way too many bad ones. I believe this has a lot to do with the fact that often scrum is introduced to fix what is already a dysfunctional organization.

In the scrum world you have a product owner representing the business side who negotiates with the software development team about features and a self organized team delivering ‘business value’ as fast as they can with a scrum master that acts as a facilitator. In this happy utopia everybody is happy and productive. This works well under the following assumptions:

1. The product owner has a firm grasp on domain and the technology.
2. The team can self organize and there is a balance between team empowerment and product owner power.
3. Everybody involved understands what they are supposed to do in their scrum role
4. The amount of work is small enough to be handled by one team

Actually, if all of the above is true, you will have little need for process. If that isn’t the case, no amount of process is going to fix things magically. In my experience, all of these assumptions are generally broken in scrum organizations. The reality is that product owners are rather well aligned with management and whatever they say ends up being an executive order. It’s elementary corporate picking order. Additionally, big organizations have many teams and consequently many product owners. Scrum in combination with a full blown corporate rat race like that is going to have some very obvious consequences. Including management not agreeing amongst themselves what needs to be done and fighting over the heads of individual teams about features, products, vision and direction. Ultimately this results in mediocre products and failure in the market.

Teams self organize to do whatever the hell they are told to do by the product owners, who with the best intentions are probably demanding the wrong things to be built on unrealistically short notice due to a fundamental lack of a grasp on either the domain, the technology, and the basics of software project management. The scrum masters basically get the shitty job of keeping it together and getting the teams to ‘democratically’ agree on stuff while not actually having any decision power to make them do the right thing. Generally the scrum master should be one of the more senior team members that would in a traditional organization have been either a project manager (with full power) or a tech lead/architect with decision powers.

I’ve seen some spectacularly mediocre software delivered this way. It’s not pretty. The problem is usually a lack of experience that gets amplified by the process. A two day pep talk about scrum is apparently all the training you need to become a product manager or scrum master. I know because I’m a certified scrum master (not practicing it though) and I know how hollow that phrase is.I congratulate Ken Schwaber on his well oiled business of scrum training large parts of the industry (me included) but I believe that he doesn’t honestly believe a two day training is enough either.

The role of product owner is generally a first step up the corporate ladder. In other words, there are a lot of inexperienced people who get promoted to be a product owner or worse start their first or second job being one. Likewise, the scrum master role is not considered a job very high on the corporate ladder either. Think developers with a few years of experience who suddenly find themselves moving post its around on walls rather than doing the stuff they were trained to do (i.e. software development). How can you expect inexperienced people like this to succeed?

Well you can’t and that is why scrum isn’t working. You can’t fix a dysfunctional software organization by rolling out scrum and mass promoting people to be product managers or scrum masters. It doesn’t work this way but yet this is exactly what happens when big companies change process. It works as follows: you spend a gazillion of dollars on training and consultancy to force the organization in a particular direction. Naturally, you want results fast so you cut some corners. And before you know it the walls of your offices are plastered with colorful post-its and the meeting rooms are full of busy looking people doing retrospectives, sprint plannings, estimation sessions, etc. Reports on velocity and story points get exchanged and proud product owners brag about their products. It’s no wonder executives love scrum: they get the illusion of progress.

The problem is that all you have at this point is a lot of people going through the moves of a process without actually delivering any coherent results whatsoever. Getting one team to deliver something is relatively easy. Getting 50 teams to deliver something coherent together is a different matter. The natural reflex in big organizations is to organize hierarchically and in a scrum context this means that product managers report to super product managers who report to department managers who report to VPs who report to the CEO who wants results yesterday. So when problems happen, they are escalated up and translated into executive orders that travel down. When it comes to solving technical problems, having a lot of non technical people involved doesn’t necessarily make things easier. So, the organization becomes a bottleneck for doing things the right way. Technical issues are solved relatively easy if you stick the engineers involved in one room. But that doesn’t happen if you add layers of management with conflicting interests that need to approve, be involved, escalate, etc. Even getting them to understand the problem is hard.

Another problem is that teams have dependencies. Team A needs the stuff of team B who needs the stuff of team C, etc. Any good software architect knows how to manage dependencies between components. But your average scrum product owner is not an architect. So, you get organizational complexity (think circular dependencies between teams) and organization driven architecture where teams raise fences (APIs, separate deployment processes, etc.) to shield them from neighboring teams. To make things worse, scrum is all about leaving teams alone during their sprints so you have a lot of teams delivering stuff every sprint and throwing it over the fence to other teams at the end. Orchestrating the development of a few dozen scrum teams is a hard problem and I’m pretty sure very few organizations have actually solved this problem. Meanwhile all the scrum training seems to be geared towards teams operating in isolation.

Is there a happy ending here? Sure, eventually inexperienced people become experienced people. Experienced people tend to do things that they know have worked for them in the past and challenge things that they know to be flawed. This includes taking control, leading and doing things in a way they believe will deliver results rather than in a way that somebody else claims will deliver results. The fallacy of scrum and agile is that initially you have a lot of people in the latter group and very few in the former.

Over the past seven years, I’ve seen scrum penetrate the minds of senior executives who then pushed for corporate wide rollout of this great new process (a well known anti pattern for scrum that Ken Schwaber warns for in his courses) that would magically fix things. Lots of product owners and scrum masters were hired/trained/promoted and then the obvious things started happening: large amounts of teams delivering the wrong software too late and not aligning and synchronizing with each other at all.

But then other things started to happen. We got some veteran, experienced software developers in some key positions in the company. They started fixing things; they started removing bureaucratic obstacles, dealing with the technical debt, and they started emphasizing good practices such as accountability, technical skill, and favoring pragmatism over dogmatism. I would say by this point, scrum is definitely on the way out. We still have floors where the walls are still plastered in post its. But we also have other floors where the walls are now empty but are still bearing the traces of what used to be scrum boards and where engineers work on delivering great software without too much interference from random management. That’s fine. Choosing and tuning the way in which you work is more important than having everybody mindlessly follow the same process.

The story of Nokia (where I work), which is often cited as an early adopter of scrum, has been well publicized and clearly, we’ve had a huge software problem over the past years. After a few bad years, Nokia is now delivering good, solid products again. The recent product announcements like the Lumia phones and Nokia Drive are good examples of a software development organization delivering results fast (i.e. truly agile). Few people appreciate how little time has actually passed between the initial announcements early this year and us delivering the goods this summer. Getting any organization this large too deliver that fast is kind of impressive. I’m afraid I wouldn’t give scrum much credit for it.

In the end what matters is having a good mix of people with experience, skills and vision. Inexperienced people learn quick in such an environment and good people get rewarded for good work.

The main problem I have with this is that these people are confusing problems and causes here and effectively blaming a solution they have for causing a problem they have. Roughly the argument goes as follows: feature branches cause people to accumulate change that then becomes very disruptive when it lands on the main branch. There will be CI breakage and lots of problems for people on other feature branches that are suddenly faced with merge issues. I think Martin Fowler actually gets it but some of his followers seem to be confused. By all means, feature toggles are a great way to get changes in early and have them exposed. It’s a powerful tool that you can use to get changes out earlier. Early is good: use it. However, it is not the case that if you use feature branches there has to be a lot of pain for everybody. In other words, feature branches are not inherently evil and don’t have to be problematic.

It’s not the practice of feature branching that is the problem but the fact that testing and continuous integration are not decentralized in a lot of organizations. In other words until your changes land on the central branch, you are not doing the due diligence of testing. Even worse, you are not making sure you have tested your changes before you add them to the main branch.

You can’t do decentralized versioning unless you also decentralize your testing and integration. Git has value when used as a SVN replacement. Git has more value when used as a DVCS. There is no good reason why you can’t do decentralized testing and integration with git. Rather the opposite: it has been designed with exactly this in mind. The whole point of git is divide and conquer. Break the changes up: decentralize the testing and integration work and solve the vast majority of problems before change is pushed upstream. If you push your problems along with your changes you are doing it wrong. Decentralized integration is a very clever strategy that is based on the notion that the effort involved with testing and integration scales exponentially rather than linearly with the amount of change. By decentralizing you have many people working on smaller sets of changes that are much easier to deal with: the collaborative effort on testing decreases and when it all comes together you have a much smaller set of problems to deal with.

This is how the linux kernel manages to remain stable, despite the fact that the amount of change there is measured in thousands of LOC/day. If you need proof that thousands of people can work collaboratively on millions of lines of code using git branches: look at the linux kernel. The amount of change, integration and testing effort, etc. in whatever you are working on probably pales in comparison with that: your problems are easy.

Here’s a few simple practices that will address most of the issues:

1. No change that will break CI on the main branch is allowed on the main branch. Zero tolerance on this one. In git terms: rebase against main, run ci test on the feature branch, fix any problems, push. You can automate this even: jenkins pretty much supports this out of the box. If main breaks ever, somebody doesn’t get the basics: educate them with a big clue bat. Rationale, it is vital to keep main stable at all times. That way everybody on a feature branch will know it is safe to rebase.
2. Rebase against main frequently, especially if you do big changes. Rationale: you are doing the changes, it is you that will take the pain of doing the integration work when things go wrong, not everybody else. The earlier you know about problems, the easier it is to fix. Feature branches are not about stopping rebases. If your feature branch is way behind, you have done something very wrong. Don’t ever do that without good reason. If it is on the central branch, you will have to deal with it at some point: so get it over with ASAP.
3. Commit frequently, keep commits as small as you can. Rationale: smaller commits are easier to analyze and fix when you have conflicts. Also, Git is really good at applying commits one by one. If you isolate the merge problems to a handful of commits, rebasing is pretty much painless.
4. Push as early as you can. If the CI builds are green on your branch and you are confident things are fine, push and don’t wait. Don’t accumulate integration work for others. Feature branches are not about hiding change but about isolating change. There’s a difference. One is a communication problem and the other is a proven strategy of divide and conquer. If appropriate, feature toggles are indeed a great way to land experimental changes. I believe this was the main point Martin Fowler was trying to make.
5. Communicate clearly around big code restructurings. Rationale, everybody rebasing against your changes will experience some pain. You are causing people to have to do work, so tell them it is going to happen. I always ask people to push their changes before I push my big changes. That way, I can fix the integration issues on my side before I push.
6. Collaborate with people by pushing changes back and forth without involving the main branch. Git format-patch is your friend: you can do this by email. If somebody needs a change before you land it, you can give it to them.
7. Be aware of the cost of things. Any time people spend on things like resolving merge conflicts, doing rebases, etc. costs you. Inevitably when you branch you accept that there is going to be some cost. Keeping a branch alive means you add cost. Don’t do it needlessly. So, branch and do what you have to do and then rebase, push and delete the branch. And again, not committing is effectively the same as branching in git. It has the same cost and risk attached to it.
8. Don’t push branches to origin unless they are going to be long lived and need to be worked on by multiple people. For simple work, keep the branches local to your machine. If you are going to be doing all of the commits and integration work, the only valid reason for pushing it upstream would be backups. There are alternatives to backing this way.
9. Beyond a certain team size, the stable branch needs to be protected. Pull change rather than pushing it (this is a severely underused git feature). Junior on the third floor says his patch is ready: pull it, have a look at it and give him feedback but don’t allow him to push and bypass checks and balances. Push only works in small teams. Pull forces people to communicate.

When will (feature) branches get you in trouble? Antipatterns:

• You are working on a big change. You haven’t updated for days. Do I need to spell it out? That is just wrong. Update!
• You are pushing a big change, all the CI builds go red. Oops, test before you push you dumb idiot. You deserve all the angry looks you are going to get. If this happens a lot, consider setting up your CI environment for having a stable branch for tested commits and an unstable branch for incoming commits that need to be tested. Jenkins supports this and it will keep unstable change out of stable.
• You are doing a big change, somebody else is also doing a big change. You find out about that when you rebase and spend hours dealing with merge conflicts. Seriously: communicate before you do anything drastic and give people an opportunity to get their changes in before you ruin their day. Pushing massive changes that you know are going to cause problems when people rebase is a very egocentric thing to do. Be a team player.
• You are working on a big change. You haven’t updated or committed anything for several days on your local branch. You are effectively using your local branch as a feature branch: everybody who is not pushing change is effectively on a feature branch. Not committing is NOT a strategy to avoid using feature branches. Also, you are not committing? Why???? What’s your excuse for not committing to your own local branch? Seriously, consider using version management and stop treating git as a file server for code backups.
• You are working on a branch for an extended period of time but your CI builds only run on the central branch. Congratulations, you have just tossed out CI as a good practice. Fix it. Either have the discipline to run the CI tests manually on every commit to the branch or set up a CI build for it. Either way, don’t break the feedback loop you get with CI.

Now at this point I have to admit something: I don’t use feature branches a lot but I do tend to accumulate a few commits locally before I push them. Also. we haven’t set up stable and unstable branches in jenkins (yet, planning to). We have the occasional breakage of our CI builds. I’m actually guilty of breaking some of the builds myself. The reason/weak excuse is: I’m having a hard time changing people’s way of working. You turn your back and people stop committing and continue treating git like they are using cvs. But I’m at least aware what the real problem is here: not the fact that I have branches but the fact that our testing needs to be decentralized.

The best feature is support for multiple header photos. So I’ve cropped and uploaded a few photos from my collection and the site will now randomly pick one of the photos listed below. So simple, but so nice to have. Otherwise the theme is pretty basic, which I think is appropriate for a blog. Also it does microformats (which was a major annoyance with the old theme). Finally, it just looks nice and it is likely to be maintained for a while (with it shipping with wordpress and all).

I’ve just uploaded a weekend project to github. So, here it is, jsonj. Enjoy.

If you read my post on json a few weeks ago, you may have guessed that I’m not entirely happy with the state of json in Java relative to other languages that come with native support for json. I can’t fix that entirely but I felt I could do a better job than most other frameworks I’ve been exposed to.

So, I sat down on Sunday and started pushing this idea of just taking the Collections framework and combining that with the design of the Json classes in GSon, which I use at work currently, and throwing in some useful ideas that I’ve applied at work. The result is a nice, compact little framework that does most of what I need it to do. I will probably add a few more features to it and expand some of the existing ones. I use some static methods at work that I can probably do in a much nicer way in this framework.

Here is some example usage (note, this will likely not stay in sync with the code in github, check there for latest version):

Updated to version 0.3, check here for latest version

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113

/**
 * Copyright (c) 2011, Jilles van Gurp
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 */
package org.jsonj;
 
import static org.jsonj.tools.JsonBuilder.array;
import static org.jsonj.tools.JsonBuilder.nullValue;
import static org.jsonj.tools.JsonBuilder.object;
import static org.jsonj.tools.JsonBuilder.primitive;
import static org.jsonj.tools.JsonSerializer.serialize;
import static org.jsonj.tools.JsonSerializer.write;
import static org.testng.Assert.assertTrue;
 
import java.io.IOException;
import java.util.LinkedHashMap;
import java.util.LinkedList;
 
import org.jsonj.tools.JsonParser;
import org.testng.annotations.Test;
 
/**
 * Not really a test but a nice place to show off some how to use this.
 */
@Test
public class ShowOffTheFramework {
 
	/** this could be a singleton or a spring injected object, threadsafe of course. */
	private final JsonParser jsonParser = new JsonParser();;
 
	public void whatCanThisBabyDo() throws IOException {
		JsonObject object = object()
			.put("its", "just a hash map")
			.put("and", array(
				primitive("adding"),
				primitive("stuff"),
				object().put("is", "easy").get(),
				array("another array")))
			.put("numbers", 42)
			.put("including_this_one", 42.0)
			.put("booleanstoo", true)
			.put("nulls_if_you_insist", nullValue())
			.put("a", object().put("b", object().put("c", true).put("d", 42).put("e", "hi!").get()).get())
			.put("array",
				array("1", "2", "etc", "varargs are nice")).get();
 
		assertTrue(object instanceof LinkedHashMap, "JsonObject actually extends LinkedHashMap");
 
		// get with varargs, a natural evolution for Map
		assertTrue(object.get("a","b","c").asPrimitive().asBoolean(), "extract stuff from a nested object");
		assertTrue(object.getBoolean("a","b","c"), "or like this");
		assertTrue(object.getInt("a","b","d") == 42, "or an integer");
		assertTrue(object.getString("a","b","e").equals("hi!"), "or a string");
 
		assertTrue(object.getArray("array").isArray(), "works for arrays as well");
		assertTrue(object.getObject("a","b").isObject(), "and objects");
 
		// builders are nice, but still feels kind of repetitive
		JsonObject anotherObject = object.getOrCreateObject("1","2","3","4");
		anotherObject.put("5", "xxx");
		assertTrue(object.getString("1","2","3","4","5").equals("xxx"),"yep, we just added a string value 5 levels deep");
		JsonArray anotherArray = object.getOrCreateArray("5","4","3","2","1");
		anotherArray.add("xxx");
		assertTrue(object.getArray("5","4","3","2","1").contains("xxx"),"naturally works for arrays too");
 
		// Lets do some other stuff
		assertTrue(object.equals(object),
			"equals is implemented as a deep equals");
		assertTrue(array("a", "b").equals(array("b", "a")),
			"mostly you shouldn't care about the order of stuff in json");
		assertTrue(
			object().put("a", 1).put("b", 2).get()
			.equals(
				object().put("b", 2).put("a", 1).get()),
			"true for objects as well");
 
		// Arrays are lists
		JsonArray array = array("foo", "bar");
		assertTrue(array instanceof LinkedList, "JsonArray extends LinkedList");
		assertTrue(array.get(1) == array.get("bar"), "returns the same object");
		assertTrue(array.contains("foo"), "obviously");
		assertTrue(array.contains(primitive("foo")), "but this works as well");
 
		// serialize like this
		String serialized = serialize(object);
 
		// parse it
		JsonElement json = jsonParser.parse(serialized);
 
		// and write it straight to some stream
		write(System.out, json, false);
 
		// or pretty print it like this
		System.out.println("\n" + serialize(json, true));
	}
}

Click to see the slightly larger version.

Here’s a nice Panorama of Lisbon that I shot a few weeks ago using my new Canon 550D. It consists 10 individual 18 mega pixel photos at full focal length (135mm) to zoom in across the Tejo river (around 1500m) that were stitched together using the ever amazing Hugin. The latest version of this is quite amazing at basically figuring out everything by itself. It’s like here’s a bunch of pictures, make a panorama out of that. And then it goes off and finds common points in the photos, arranges them in a grid that makes sense. Adjusts for lens characteristics, rotates, translates and distorts the photos until they overlap perfectly, adjusts tonality of the photos to match, blends the photos seamlessly and finally spits out a big whopping tif file. There are other tools that do some of these things but this one is the best by far. If you are willing to tweak the tool, it can do some pretty funky things like HDR panoramas (i.e. stitch together overlapping photos with multiple exposures), do panoramas where the photos have different exposures and tonality, use different projections for the panorama, drag around the stitched panorama inside whichever projection you chose to influence how things look, manually identify overlapping points, plugin experimental tools and algorithms for the various steps of creating a panorama, etc.

Unfortunately, you have to down-size this kind of images for the web and apply lots of jpeg compression. The final image is 10032×1024 pixels, downsized from 21896×2235 to save some precious bandwidth. The tif weighs in at a bit over 100MB. The lowests size jpeg I managed to get out of the full image was around 2.5 MB, but at that size compression artifacts are very noticeable. So I instead re-sized to something that might fit on an over sized display (vertically at least) so that people will have to scroll in only one dimension and got the size down to 1.1 MB with a respectable compression rate of around 30%, which preserves a lot of detail.

I was positioned in front of the big Jesus statue that is visible from Lisbon. I later went up as well and shot some more photos, some of which you can admire here (along with the rest of my photos from that trip). There is one photo there with the same view in a single photo from up on the statue that gives you a nice impression of the zoom range of this camera. It looks like a panorama photo but it is really just a zoomed out photo that I cropped a little.

Anyway, happy commenting and thanks @eelkefolmer for pointing out my site was broken.

Lately, I’ve been involved in some interesting new development at work that is all about key value stores, Json, Solr and a few other technologies I might have mentioned a few times. In other words, I’m having loads of fun at work doing stuff I really like. We’ve effectively tossed out XML and SQL and are now all about Json blobs that live in KVs and are indexed in Solr. Nothing revolutionary, except it is if you have been exposed to the wonderfully bloated and complicated world of Java Enterprise Edition and associated technologies. I’ve found that there is a bunch of knee-jerk reflexes that causes Java developers to be biased to be doing all the wrong things here. This is symptomatic of the growing gap between the enterprise Java people on one side and the cool hip kids wielding cool new languages and tools on the other hand.

The problem with Json in Java

One of the things I’ve been struggling with recently is Json. Json and the java world is not a perfect marriage. The whole point of Json is that it is valid Javascript, Python, and Ruby. That doesn’t mean that you should go off and execute it but it does mean that after parsing (or rather unmarshalling), you end up with dictionaries, lists, and native types that each of those languages supports. This means you can have light weight data structures that you can just serialize to and from Json at will. In Python, classes are just dictionaries of attributes and dictionaries and lists are native types. That means that Json is a very natural serialization of any Python object structure. Javascript doesn’t have classes since it is a prototype based language, but of course lists and dictionaries are native types there as well and objects are of course dictionaries. I’m not a ruby-ist but I know enough of it that it is similar to Python and Javascript in this respect as well.

Sadly, this not the case in the Java world. Dictionaries and Lists are not native types and instead people use the Collections framework (or alternatives such as Google collections). The classes in those frameworks are of course generic these days and classes bear very little resemblance to dictionaries in Java. That means Json is not a very natural serialisation for a Java based object structure. Frameworks exist to address this through mappings, which can be convenient. Except, it reduces the Json approach to yet another way to serialise Java objects rather than a beautifully simple and elegant solution to modeling your data. And that’s where the problem lies.

My main problem with doing things using mapping is that it’s not exactly lightweight anymore. Suddenly you need to worry about instantiating classes, stuffing the right bits of data in the right setters and constructors. Then inevitably you start doing some hashCode and equals methods and of course generics creeps in somewhere along the way (or even worse, inheritance). Then you generalize and add some interfaces, annotations, etc. Congratulations: your easy to read 50 line Json file representing a simple dictionary with some very simple, uncomplicated Json magically turned into a framework with dozens of classes and thousands of lines of code that does nothing else than represent what any javascript, python, or ruby programmer would trivially accomplish with a few lines of code. That’s some serious bloat. It’s the reason countless of software engineers have been fleeing the java enterprise world in favor of more sane worlds such as Python Django, Ruby on Rails, or even Node.js. It makes me want to flee as well.

The solution: do what others do and use Json

I’ve been fighting this bloat & complexity over the past few months and have been stubbornly reducing the number of model classes we have to 0. Instead I simply use the classes provided by the GSon framework (which is one of the popular options in the Java world for dealing with Json) to represent the core Json objects, I.e. JsonObject and JsonArray + the four primitive types (int, double, string, boolean). Where in the past I would have been constructing model classes and instantiating those, I instead do a “new JsonObject()” and start modifying it through its primitives for adding JsonObjects, JsonArrays, or one of the four JsonPrimitives. This is different than what most Json frameworks in Java try to achieve, which is providing convient ways to map to and from Json and your model classes. What I want is to get rid of model classes.

You can lose some serious amounts of source code this way. People seem terribly concerned with hiding Json from the Java programmer by adding layers of indirection to hide it. What they don’t realize is that there is a direct correlation between lines of code and complexity on one hand and maintenance cost on the other hand. Json world: you are adding string “foo” to a dictionary. Java world, you are creating an instance of class Foo<String> and another of FooHolder<Foo<String>>. Then you add your instance of Foo to the FooHolder and then you serialize it to Json. I’m exaggerating of course but I’ve seen people write code that essentially boiled down to setting a few properties on a dictionary and yet somehow required 10+ classes and an afternoon of coding to accomplish. That feels very wrong to me.

Problems with the Java language

The problem is actually quite subtle and strongly related to a few areas where Java is lacking relative to more modern languages:

• Java distinguishes between native types and classes (which are a native type with a shallow non native wrapper Class, but you can’t ‘instantiate’ a new class). That means for example that a class is not a dictionary and lists and dictionaries are not native types. This causes all sorts of issues with reflection and advanced typing (like generics) or the introduction of closures to the language. This is a problem more modern object oriented languages don’t suffer from since in those languages everything is natively an object, including a class.
• Static typing is nice but often dynamic typing is good enough. As can be seen see with Scala, or Mirah (a natively compilable Ruby dialect), it doesn’t have to be so black and white. Fact: the Java type system represents the state of the art in the early eighties and does not incorporate many of the advances since then. Modern languages can be far more expressive and less verbose without much sacrifice.
• Java has no good way of representing multi typed dictionaries because of its rigid type system. You cannot easily express a dictionary where key “foo” maps to an integer and key “bar” maps to another dictionary. In other words, Java lacks the language capability to express the contents of a Json dictionary elegantly. The ‘workaround’ is to use Object as the type, which then leads to a lot of casting elsewhere. Generics don’t quite address this. This is a fundamental and critical flaw that has given rise to an enormous amount of complexity in the Java enterprise world. Any time you hear Java developers talk about mappings and frameworks, they are working around this problem. Massive amounts of code are involved with working around this problem.
• The lack of proper typing and closures leads Java coders to believe they need to model the world in terms of classes and objects. This is a very serious problem because it naturally leads to complex frameworks for even the simplest of domains. If you’ve been exposed to a decent amount of functional programming, you will feel dirty after a day of work with any of the big Java frameworks. They’re big, clumsy, and tedious. And they generally don’t play nice together and fuse data and functionality together in a way that actually prevents reuse.
• Java has no first class properties and instead relies on the JavaBeans convention of having lots of setters and getters. So where in a decent language you would go object.attribute.attribute2.attribute3 = foo, you get in Java several lines of code with setter and getter calls as well as null checks just to stuff foo in the right place in the object graph.

Principles I stick to

So, does that mean I’d love to switch languages to something more modern? Absolutely, I’m bored out my mind with Java! But sadly, that’s not likely to be happening on any of the projects I’m currently involved with and I have to work with what I have. So, instead I’ve started to resist the traditional Java way since I believe a lot of these practices to be outdated and counter productive. I have started to incorporate and borrow patterns and idioms from elsewhere.

• Json is the primary way to represent data. Any data that goes in or out of the system via the API is represented as Json. That makes Json the primary representation of the data. That also means any representation other than Json is a secondary concern and as such a waste of time and resources. That includes any kind of Java class or object hierarchy intended to represent the data that is in the Json. That is a reverse of principle for a traditional Java programmer who will consider Java his most important way of expressing things and will consider json a secondary concern.
• Use a dedicated set of classes to represent Json in Java. Because of several of the before mentioned language issues, the Java collections framework is not very suitable for representing Json in Java. I tend to prefer the way it is done in the before mentioned GSon framework. So I use JsonArray, JsonObject, JsonElement, JsonPrimitive (and its four subclasses).
• Use a key value store. A key value store is nothing but a persistent dictionary. So, Key Value stores are the natural way to deal with persistence. Listen carefully, I just wiped an entire layer from the traditional enterprise architecture. That’s good. No more object relational mappings. Json is what goes into your system and it is what goes out into the storage layer.
• Resist secondary representations and stick to the primary representation: Json. Don’t do secondary representations and associated mappings, translations, frameworks, etc. because every time the primary representation (i.e. Json structure) evolves, it triggers a ripple effect of code changes related to secondary code artifacts. Avoid this unnecessary cost by not having those.
• Don’t expose your internals. Business logic natively manipulates Json objects because business logic is a primary concern. Any usage of non Json objects is disallowed in public facing APIs. So, never return List, Maps, etc. but instead return and accept JsonArray and JsonObject instances (or Java native types). Don’t force your API users into using your internal data representation. Every time you violate this rule, you are leaking internals through your API. Don’t do that, it’s bad design.
• Everything may become external some day. Of course what is internal and external is somewhat a blurry distinction. So, it follows that you must use Json internally as well since it may one day become external.
• No needless abstractions. All abstractions are leaky. This is particularly true of domain models. What seems like a good abstraction now may prove to be a horribly poor idea a few years down the road. The world is full of frameworks and domain models that don’t play nice with each other and are poorly aligned (you likely are guilty of inventing several of them). Your Json input and output is the one and only representation of the data so don’t even try to abstract from it.
• KISS and YAGNI. Keeping things small and simple removes much of the need for complexity. Each extra class is complexity, each extra method is complexity, each extra mapping is complexity, each extra layer is complexity. If your data is simple, your code should be simple as well. Don’t add complexity you don’t need because you aint gonna need it. For example, the GeoJson representation of a coordinate is an array of two doubles. I’d hate to see the equivalent of GeoJson as a Java framework. I shudder just thinking of all those tedious little classes to represent points, polygons, etc. that are of course completely incompatible with similar classes you’d find in Java2D, OpenGl, or other graphics frameworks. Don’t do it. Array of two doubles is good enough.
• Respect the law of Demeter. Json is inherently weakly typed, so don’t assume to much about its structure and dig out that JsonObject from your main Json document and then work directly on that object rather than indirectly by passing around references to the main JsonObject. This is simply applying the law of Demeter, which requires you to use no needless levels of indirection when accessing members of an object. So bad: String s = root.foo.bar.foobar.bar, right: s = foobar.bar embedded in some code that is parametrised with whatever happens to contain foobar. Modularise your code around small, uncomplicated Json objects rather than big, deeply nested ones. Separate the assumptions about which objects go where from the assumptions of what is inside those objects because both are likely to change and because those objects might start to appear in different places.
• Design your Json for more capable languages. You are not merely serialising data, you are creating a Json document that others are going to use from e.g. javascript. It is worth spending some time considering their needs and requirements. Read: they don’t like big, complicated Json that is hard to read.

It’s not all good of course and there are downsides to rebelling against the Java way:

• Refactoring becomes a bit more tedious. I tend to use constants for my attribute names, which partially addresses this. But forget about manipulating the Json structure with the refactoring tooling in your IDE. This is particularly annoying for business logic depending on that structure. A work around here is to keep things simple.
• Extracting data from Json structures can be tedious and may involve a lot of null checks. We work around this with a few static helper methods. Varargs is your friend here. e.g. extract(object, attr1, attr2, attr3) would return either null or the value of object.attr1.attr2.attr3. Mind you, you have exactly the same issue with many Java object graphs and this is a likely cause for any NullPointerException you’ve ever encountered.
• You will at points miss the convenience of model classes. This is the price you pay for the Java type system. Keep dreaming of a better world where you don’t have to use it anymore. The alternative of using model classes in Java is a fallacy though. It’s entirely true that you would end up using model classes in Python though. However, Python classes are dictionaries and as such need no translation work whatsoever. Java classes are not dictionaries and do require translation work.
• Like all good frameworks, the Json classes in GSon are final so you can’t sneak in model classes by extending JsonObject. However, you might consider using wrapper objects that have a delegate JsonObject as the single field. I’m not strongly against this but it still feels like bloat to me and I prefer to avoid this pattern because of KISS and YAGNI.
• There is a lot of code out there that assumes you are doing things the Java way, reusing that just got a little harder. But I again refer you to the principle on KISS and YAGNI.

Why this is important

I’ve been refactoring quite a bit of code using the above principles and I can report to have shed many lines of code along the way while improving readability, reducing complexity, and gaining flexibility. My background as a software engineering researcher tells me this is a good thing (TM) because lines of code directly correlate to software development cost, complexity, and maintenance cost. Therefore, less lines of code and complexity is almost always the right thing to aim for unless you can convert the investment in cost into added value. More lines of code is not necessarily the same as adding value. If it is not adding value, it is in fact technical debt.

I’ve seen a lot of Java projects lately that are needlessly complicated and hard to maintain while trying to achieve seemingly simple goals that involve accepting some data, stuffing it in a data store, retrieving it and and returning it via some API. CRUD in other words. CRUD is simple but not if it has to go to several layers of indirection that involve mapping xml to data transfer objects and then to model objects, and then to SQL tables and back again. The system we are replacing with a new system based on the principles outlined here does this. Its business logic amounts to take in data, validate it, store it, mash it together in an interesting way and retrieve it. Its business logic should also include a bunch of other things that we never got around to because we got bogged down in non value adding activities related to mapping data in different ways and dealing with over engineered & bloated technologies that got abused in a terrible way by some junior engineers who thought they were in a candy store. In other words, I’ve gone through months of head shaking, swearing a lot, silently cursing misc. individuals no longer on the team, and mostly deleting code to replace it with small amounts of new code. In other words, I’ve taken a system that pretty much A** rapes the listed principles and am working hard on building a replacement that doesn’t do that.

Once upon a time, I signed up on the Adobe site to use the online version of photoshop. I probably was still in Finland at the time so naturally, I used their English localized web site. Fast forward to today and I am interested in downloading the trial version of Adobe Lightroom to play with it and possibly buy it.

To do that you need to log in. So I log in. Then it asks me for my address. Fine. Except, I don’t live in the US and there is no country switching thingy in the form. Apparently it hardwired the country choice (which really was a language preference) into my account. Fail #1.

It gets worse. They do have a country switcher on the front page and in the store. So I set that to Germany. Now it addresses me in German. Fail #2

Of course there is no language switcher. It is assumed (wrongly) that I must be a native speaker and therefore want to be addressed in German. Always. Fail #3

So by this point I decide to be pragmatic and select some region they haven’t localized for to get the site in English. So I select Eastern Europe, which is apparently a region not worth breaking up into countries and technically I live in the former DDR, which used to be eastern Europe. So I once again go to the store. Prices in euros this time. Great! Although effectively the price just went up 80 euros since 300$ is ~220 Euro and not 300 Euro! Then I get to the point I need to enter my address, it still insists on a US address. Fail #4. If people select a region, it’s probably because they live there.

Underlying this problem is a common failure of online retailers to grasp the basics of internationalization and demographics that is causing them to lose the business of millions of potential customers who have a language/region combination that they can’t support. This is especially silly when the language is English and the online retailer is US based. English is a great language. World wide, there must be many more non native speakers of it than native speakers. Especially migrants (like) me tend to rely heavily on English.

So, I am not going to be an Adobe customer.

I’m not an iTunes user either for the same reason and for the reason that I’m not big into German dubbed movies and schlagers, which is pretty much what they tried to sell me on the few times I tried to use it. Also, I declined to accept the terms of use when I was living in Finland because they were in Finnish, which is something I don’t speak and I tend not to agree with official looking text that I can’t read.

Does it have rough edges? Sadly, yes there are some. But no major show stoppers. It’s stable. The camera is awesome. The video playback is great. You can play most movies without re-encoding them. I managed to not run out of battery during a trans atlantic flight, watching movies non stop, and the oled screen is fantastic.

I’m particularly happy with how maps is improving with each version.

It went from something that barely worked a few years ago to being arguably one of the best mobile maps experiences around. It’s certainly very competitive when it comes to map quality, navigation, and offline usage.Disclaimer, I work on the backend services for this application .

I’ve used two ovi store applications to write this post. One is called swype and it replaces the touchscreen keyboard with a version that allows you to ‘draw’ across the keyboard and it figures out the words with great accuracy and speed.

The other is the official wordpress client, which I’m trying for the first time, and of course that’s the real reason for writing all this .

So far so good. I managed to write a good chunk of text with swype and the wordpress app seems to do everything I need it to do.

All together this is a pretty good phone and it is getting even better with upcoming software releases. I think the recent focus of nokia on software quality and end user experience is starting to pay off. It’s not without flaws of course and it’s definitely to early to call the battle with our competitors over but it is progress and I think we’re getting there.

A final note here. I work for Nokia and am of course slightly biased. That being said, I rarely review stuff we produce and as a rule only when I really like it, without reservations. I loved the n900, which I reviewed a few months ago, but I’ve had several nokia phones that I chose not to review as well for this reason.

For those who got to know me more recently (or don’t know me at all). I have been moving back and forth between academic jobs and non academic jobs for about twelve years. The last two years I’ve been employed in a strictly non-academic role, which I enjoy very much. Before that I was working (and publishing) at the Nokia Research Center. In that job, I squeezed out several not so important workshop articles and book chapters as well as two major, big journal articles (which in the academic world is all that counts). Those articles took a long time to write and even longer to get reviewed, edited, re-reviewed, re-edited, accepted, re-edited, approved, edited by a professional editor, approved, pre-published, and finally printed.

Practical Web-Based Smart Spaces
Abstract:

Mobile devices are evolving into hubs of content and context information. Many research projects have shown the potential for new pervasive computing applications. This article shows how Web and resource-based smart spaces can support pervasive applications in a wide variety of devices. A framework that employs a resource-based HTTP style for pervasive services called Representational State Transfer (REST) can enable easy mashup of applications. This framework has several important features. First, a flexible access control mechanism on top of the OpenID and OAuth protocols provides security and access control in heterogeneous, dynamic environments. Second, a search engine can collaborate with existing service and network discovery mechanisms to find resources on the basis of their indoor location. Finally, an emerging W3C standard, Delivery Context: Client Interfaces (DCCI), facilitates sharing information within a device in an interoperable fashion.

The first article, we decided to write in 2008 as a way to promote the fine research we had been doing in Nokia Research. You can’t get better marketing for your research than an article in a high profile magazine with a wide distribution in the research community. The magazine we selected for this was Pervasive Computing, and I’m very proud and happy we got our article in. Since this is a magazine and not a journal, the article is comparatively limited in size, which means that it posed some interesting challenges on what to keep in it and what to omit.

Comparing Practices for Reuse in Integration-oriented Software Product Lines and Large Open Source Software Projects
Abstract:

This paper compares organization and practices for software reuse in integration- oriented software product lines and open source software projects. The main observation is that both approaches are successful regarding large variability and reuse, but differ widely in practices and organization. To capture practices in large open source projects, we describe an open compositional model, which reflects their more decentralized organization of software development. We capture key practices and organizational forms for this and validate these by comparing four case studies to this model. Two of these studies are based on published software product line case studies, for the other two we analyze the practices in two large and successful open source projects based on their published developer documentation. Our analysis highlights key differences between the practices in the two open source organizations and the more integrational practices used in the other two cases. Finally, we discuss which practices are successful in which environment and how current practices can move towards more open, widely scoped and distributed software development.

I started writing the second article, already in 2006. The first drafts were not very satisfying and we put it on ice for quite some time until deciding to finally get it published in 2008, which meant a more or less full rewrite of what we had until then. From there it took until September 2010 to get it printed. Most of those 2+ years were spent waiting and very occasionally doing major revisions of the article in response to some reviews and editor comments.

This last article (for now) has some continuity from my earlier work on software variability, software product lines, and software design erosion that I covered in my Ph. D thesis in 2003 (and several related publications). We present a model for large scale software development that we reconstructed from observing “how it’s done” in several case studies published by others as well as in the open source community as well as our own experience studying various systems and companies, as well as getting our own hands dirty with actual software engineering. Two years of subsequent practicing real software development in Nokia has only strengthened my belief in the vision presented in the paper, which is that the only proper way to scale software development to a software eco system (i.e. a thriving community of many developers across many organizations working with and on the software) is to decentralize management of the development process. If you are interested in this and want to read more, co-author Jan Bosch, my former Ph. D. supervisor who now works at Intuit, has been publicizing this view as well in his frequent talks and keynotes at various conferences. This website is dedicated to this topic.

Now with both articles out of the way, the question is “what’s next?”. The answer to that, for now, is nothing because I haven’t started writing any new articles in the last two years. And frankly I’m not likely to start writing one soon since I lack the time and I’m appalled by the snail pace at which the publication process progresses. Both articles have turned out very nice but would have had much greater impact if we had been able to get them written and published in the same year instead of (nearly) 3 to 5 years apart. Sadly, this is the reality of academic life. You write your stuff, you move on, and some day stuff actually gets printed on dead trees. There are more reasons, which I won’t rant about here but it is a big factor in me not being interested in publishing any more, for now.

Anyway what pissed me off was the following:

On Sunday morning/night (jet lag), I called the phone number listed on my bank card to find out what the hell is going on.

This lands you in the standard useless menu where some very tedious computer voice slowly guides you to your destination repeating every choice you make and asking whether that was really what you wanted. This one was particularly annoying because they insisted on my full account number (they had speech recognition though) and branch number of my local bank office, which is apparently very important in Germany. Naturally, the first questions I got when I finally got a human to talk to me were what my branch and account number were.

Then the misery started, nobody on the other side of the line could do anything useful for me. Lady #1 who luckily spoke English connected me to Lady #2 who didn’t and quite rudely told me to go away (in German). So, I had to go through the above mentioned menu again to talk to Guy #1. I then asked him “do you speak English” and he said yes but added “I’m not allowed to speak to customers in English”. Eh right. He clearly detected my anger/amazement and then proceeded in English anyway, for which I’m grateful, I guess. He then figured out that I probably wanted to talk to someone in my branch office. So he gave me a number for Lady #3 who of course would not be reachable until Monday. So, I was stuck for another 24 hours without cash on me and I still had no clue why my card was not working (e.g. somebody might have skimmed my card and is emptying my account).

Before getting started, make sure you have both git-core and git-svn installed (ubuntu) or that you have done a “sudo port install git-core +svn +bash_completion” if you have a mac. Also, you probably want to configure git properly. Google is your friend, there are plenty of tutorials on the basics of git and svn that you might want to run through first.

Both git and svn use urls to identify repositories. For the sake of simplicity, I will use a very basic example involving three local repositories on my file system with file:// based urls but of course you can use http:// ssh+svn:// or git:// type urls just the same:

• file:///Users/jilles/test/svnrepo – A little svn repository. Generally this will be your central svn repo with years of work inside it.
• file:///Users/jilles/test/git1 – A git repository that we’ll create using git svn.
• file:///Users/jilles/test/git2 – Another git repository that we will create using git clone.

Lets start by creating a simple svn repo, checking out a svn work copy, and committing some stuff using svn:

cd /Users/jilles/test/
svnadmin create svnrepo
svn co file:///Users/jilles/test/svnrepo svnwc
cd svnwc
echo "hello" > hi.txt
svn add hi.txt
svn commit -m'hi.txt added'
svn up

Ok, we now have a svn repo with some stuff in it. In the git world, you always work on a local repository. You don’t check out from a remote repository, instead you clone it. So, lets git svn clone it to a local git repository and make some changes there and then commit it back to the svn repo:

cd /Users/jilles/test/
# clone the svn repository to a git repository using git-svn ...
git svn clone file:///Users/jilles/test/svnrepo git1
cd git1
# observe there is a hi.txt and a .git directory. here
echo " World" >> hi.txt
# stage a change. 
git add hi.txt
# now commit the change to your local git repository (i.e. your .git directory contains it)
git commit -m"say hi to the world"
# and finally push the local change back to svn
git svn dcommit

Now lets check on the svn side if it worked …

cd /Users/jilles/test/svnwc
svn up

Yup, our change is there.

Now imagine a co-worker comes along, who wants to work on his own git repository: git2. To get started, he clones the existing git1 repository and right away makes some changes.

cd /Users/jilles/test
# lets not use git svn and instead do a regular git clone, either way would be fine btw.
git clone file:///Users/jilles/test/git1 git2
cd git2
echo '!' >> hi.txt
git add hi.txt
# commit the change to the local git2 repository
git commit -m"finishing touch"

Back to git1, who is of course interested in the change made in git2:

cd /Users/jilles/test/git1
# add git2 as a remote. 
git remote add git2 file:///Users/jilles/test/git2
# fetch tags and branches known on git2
git fetch git2
# pull the change made on git2 into the master branch (on git1)
git pull git2 master

Git1 and Git2 now have the same change but it is not in svn yet. Git2 can’t commit to svn yet. So, lets fix that. Open /Users/jilles/test/git2/.git/config in an editor and append the following section:

[svn-remote "svn"]
        url = file:///Users/jilles/test/svnrepo
        fetch = :refs/remotes/git-svn

Save the file and:

cd /Users/jilles/test/git2
# fetch the svn branch and tag information
git svn fetch
# make sure we have all the latest changes on svn by "rebasing"
git svn rebase
# now we are ready to dcommit the change we made in git2 back to svn
git svn dcommit

Good, the change is now back in svn. Lets verify and modify some more in the svn world:

cd /Users/jilles/test/svnwc
# get the latest changes from the svn repo
svn up
# fix the file once more ...
echo 'hello world!' > hi.txt
# and commit it
svn commit -m"stupid newlines"

And lets rebase git1 and git2 against svn:

cd /Users/jilles/test/git1
git svn rebase
cd /Users/jilles/test/git2
git svn rebase

Now git1, git2, and svn are in sync and have the same history. If you examine the history with git log on git1 and git2, you will notice that all the commits have the same hashes. This is no coincidence, commits have globally unique sha1 hashes that are based on their content. The same changes will yield the same hashes no matter where you make them. This is the reason git works so well. It doesn’t matter where the change came from, as long as the hashes line up, everything will be fine.

I’ve demonstrated that git1 and git2 can pull changes from each other and that you can rebase and dcommit against the central svn repository. You needn’t be afraid of double commits or lost commits. Git all sorts it out. From git’s point of view, a central svn is just another repository. Git is decentralized and throwing svn in the mix doesn’t make much of a difference. It just integrates more or less seamlessly. So, you and your git using co-workers can go off and do all the cool stuff git users do, while all staying in sync with the central svn repository.

Random things you probably want to play with next:

• git svn clone your svn repository; or git clone it from one of your git using colleagues and manually add the svn remote
• use git daemon to set up a network daemon that your colleagues can pull changes from remotely
• create a local branch and integrate changes from your colleagues there before merging back to your git master and dcommitting the change to svn
• allow others to pull your experimental branch

I have over the past year been a bit less active blogging and posting in forums. I make an exception on things related to build processes, distributed version management and Java and you may find a couple of fine rants on my website and across various blogs on this topic.

This post is intended to pull all of it together in a bit more grand vision and reflect a bit on what a post revolution software engineering world might look like. I don’t like or believe much in revolutions but I do recognize that getting from where we are to where I think we need to be amounts to one. The more likely outcome is an evolutionary development where the bits and pieces where I was not completely and utterly wrong might actually happen. At some point. Maybe.

The Problem: Files

Software engineering practice is stuck in abstractions and concepts dating back decades. Because it is stuck, it is not really moving forward as fast as it could be towards being more capable of dealing with scale in numbers of people, amount of software, and amount of evolutionary change per time unit. This annoys me and has done so for years. It is the main motivation for writing this article.

The core abstraction I strongly believe is very wrong is the file. A file is a binary or text blob that lives on a file system that has some name. It has some content of some type that is generally associated with a suffix in the name (e.g. *.java).

Software engineers mistakenly believe their output consists of software artifacts that are stored as files. It doesn’t. Their output consists of software that happens to be stored in files. If you were born in the fifties, you’d be serializing your software from your head onto paper and manually flip switches to de-serialize. This was tedious but it worked. People like Ada Lovelace were writing software long before there were machines capable of even executing the software (other than the human brain, which is not very good at running our own software). The sixties made IO quite a bit easier with consoles, punch cards and tapes. People would transfer programs to and from RAM buffers by streaming them from tape or by feeding piles of cards to a computer. Somewhere in this era the notion of transforming these software data blobs using other software blobs also became popular: I’m talking about editors, compilers and interpreters. Finally somewhere in the seventies floppies and hard disks started to be used and everybody started putting their data blobs in files. Sadly this is where evolution stops.

At least when it comes to storing computer programs. For other types of data we have gotten a lot smarter. We have graph databases, object databases, tuple stores, relational databases, document stores, etc. None of these are commonly used to store computer programs.

This is weird and in my view it is holding software engineering practice back. A big reason is that we have built a huge sand castle of technology around the notion of files as the primary means of storing computer programs. For example we have version control systems that store different versions of files. We have built tools that consume and produce files. We have source code analyzers that expect to be working on files. We have IDEs that offer us convenient ways to edit files. Once we stop putting software in files, that technology needs to start changing as well.

Smalltalk and Eclipse

Actually it is not strictly true that evolution stopped with the file. Smalltalk and its descendants have a very interesting approach where the entire software system is stored in one big image file. Effectively smalltalk is using a database concept. This allowed for some interesting innovations, like meta programming and software transformations (aka refactoring) that require some readily available in memory representation of the software that is generally very rich and graph like (i.e. an abstract syntax tree) and can be introspected. Smalltalk developers realized that it didn’t make much sense to fragment bits and pieces of the software over multiple files. You might as well just use a single file.

This inspired IBM to create a nice IDE for Java at some point: Visual Age. Visual Age was very much a reborn small talk style IDE and crucially stored its Java code not in files but in the form of a database that effectively was a serialization of the abstract syntax tree of the entire software system. It needed that tree in order to be able to power important features like refactoring, code browsers, etc. Later it became Eclipse and they dropped the notion of storing the serialized AST opting instead for a combination of on the fly construction of an in memory AST, on the fly compilation and analysis, and reading to and from files the good old fashioned way.

Failed experiment: intentional programming?

Some late nineties papers by Charles Simonyi on Intentional Programming and persistent rumors about him actually being very close to launching related products was about taking the whole smalltalk/visual age thing to the next level. It’s too early to call this a failed experiment because Simonyi never really delivered the goods. His company (Intentional Software) is still hyping intentional programming but has yet to ship a product. Seriously, this has been in the making longer than Duke Nukem Forever. In a nutshell Simonyi’s very brilliant idea is that creating software is about coming up with abstractions that are represented in the form of abstract syntax trees that can be translated into other, more general abstractions in multiple iterations until you end up with a syntax tree that can simply be serialized to executable code. His core idea was to treat the transformations and not the abstractions as the first class entities. In a intentional programming world you start with really simple abstractions that you can translate into executable code and you build increasingly more complex and specialized abstractions that can be used for specialist or domain specific things. The traditional notion of compiling is very similar except it is a bit limited in the number of transformations and the abstractness of the abstractions involved. Basically most languages go to roughly 2 or 3 transformations: source code to abstract syntax tree to assembly to executable bits and bytes. There are lots of variations here but it is essentially a pipe line. Intentional software’s product works differently. The user manipulates the AST of whatever abstractions he’s working with in some rich editing environment that presents appropriate views on the AST and tools to edit the AST. Applicable transformations and optimizations kick in on the fly to keep the executable code up to date. It probably does a bit of serialization here and there but probably not in a format that is very notepad friendly.

Another failed experiment: Model Driven Architecture (MDA)

Readers of the previous may be tempted to sweep this in the same bucket as Model Driven Architecture (MDA). MDA was something that emerged out of the lovely world of UML. In MDA one defines systems using a standardized meta language, generally inside a dedicated tool (e.g. Rational Rose). The idea is on the surface very similar to intentional programming except it is more artifact centric. The weak spot of MDA has always been that the transformations from models to software is kind of a hacky process that is generally locked up in some proprietary tool. Basically there is the meta language and the UML language (of course defined in the meta language) and UML models. Where the whole thing becomes messy is the transformation to actual software. This bit is more or less completely proprietary and your mileage may vary though apparently some tools are quite good at this. Early MDA environments were monstrous tools with loads of J2EE, application server madness and a rich sauce of UML and XML spread on top. It’s sort of a lot of not so great things stuck together and people have wasted enormous amounts of cash on making sense of the whole thing, which continues to feed hordes of very expensive consultants. MDA never really dragged itself outside the domain of finance and banking applications.

There is a lot of hype currently around language benchmarks and similar technologies that is partly fueled by Martin Fowler who is about to publish a book on this and other topics related to Domain Specific Languages. He also has several articles worth reading on his website on this topic.

The point about ASTs

The whole point about bringing up MDA and ASTs is that software is made out of abstractions. The serialization of abstractions to and from source code in e.g. C or Java is lossy in the sense that some of these abstractions are no longer explicit when the serialization is complete. For example you might have a nice program that implements the visitor pattern, which is an abstraction many OO programmers would understand. This might be obvious to the reader of the source code and it might have been documented in some comment but it is no longer explicit whereas when the developer had the design in his head, or on a white board, or possibly even in a UML diagram, it was explicit. Intentional programming and MDA are about giving developers the option to keep those abstractions around and to define their systems in terms of richer abstractions than are found in typical programming languages.

The point about files

A useful representation of the abstractions is the abstract syntax tree, which is an internal data structure used by compilers, IDEs, source code analyzers, etc. There are of course many interesting ways to serialize tree like data structures like ASTs. For example, XML is pretty popular (though not for its readability or editability). And of course source code is nothing more than the serialization of an AST that conforms to a particular grammar. The point about files however is that they are used as containers of abstractions. Generally you need multiple such containers to get a working system. They might be of different types even (your average J2EE architecture comes with dozens of them). Finally, the notion of a file as the boundary on a container of abstractions is rather arbitrary. In some languages you put a single class in a file. In others files are called modules and can contain multiple classes, functions, procedures, etc. Additionally files live in directories, which are containers of files. In many languages directories have semantics as well. For example Java uses nested directories that must match the package structure (which is also nested).

I’d argue that files and directories are poorly suited for modularizing your system. Also as a boundary of abstraction between modules, they are both too arbitrary and too coarse grained. For example when you consider versioning, you are stuck with files and directories even though many version systems don’t even have a first class representation for directories (git) or have trouble keeping track of name changes to files (cvs).

Development is a collaborative process

Development is essentially a collaborative process. Except for very small systems, most software needs to be developed together with other people that may even be spread all over the world in some cases. It is no surprise that there is a huge amount of technology available to facilitate collaborative development. I already touched on version control systems. In addition to that there are bug management systems; collaborative editors; source code review systems; tools to examine differences between files. If development is a collaborative process, why can’t we version, review, triage bugs, exchange, communicate, etc. via the internet. Why do I have to bother serializing and deserializing trees (aka. saving and opening) ASTs in order to do stuff?

To the point …

This brings me to the point, finally. Files don’t make sense at all in many ways, as I’ve argued so far. So, lets not use files then. Lets look at a possible alternative instead:

(Disclaimer: very likely some people have implemented bits and pieces of what I’m about to outline. I’m aware of this but still want to provide a bigger picture here.)

• Cloud storage. Software is a data structure, not a file. Data structures should of course live somewhere. These days the appropriate place for data to live would be the cloud. By cloud I mean that programmers should be accessing their ASTs via tools that are connected to the network and that automatically sync back and forth. The tool could be browser based or it could be some native UI. It might even resemble a code editor.
• Versioning and other content management features. Versioning is not a feature that is limited to version control systems. For example, many content management systems do a decent job of versioning the objects they manage, which in the case of systems like Drupal are actually trees of objects. Versioning is just one of the many useful features you can find in content management systems. Others include auditing, security features, syndication of updates, workflows, etc. There is no reason why a cloud based software store should not have these features. In fact, they might prove to be highly useful. So, naturally our cloud based AST store should come with all these features.
• Collaborative editing. Collaborative text editors have been around forever but somehow never really caught on in the software world. Also browser based office tools like Google docs and the recently discontinued Google wave have some really interesting collaborative editing features. I still think Wave could have been a really nice eclipse replacement. Naturally we want collaborative editing tools for our cloud based ASTs.
• Offline & synchronization. Being connected to the cloud is of course nice but sometimes people need to be offline (e.g. a plane) or there are disruptions in the network. In a cloud based world that could end up being highly disruptive for developers. However, there exist several good technical solutions. One of the trendy nosql solutions out there, couchdb, is all about storing stuff in a decentralized way and syncing over http. Ubuntu is essentially the worlds largest data store where each of its users has a private couchdb instance that (optionally) syncs with an Ubuntu provided central couchdb store. Yesterday a company called couchio announced a couchdb based product, CoucheOne, that is specifically designed to address data synchronization on mobile phones. The point being here that even though software needs to live in the cloud, it should be possible to go offline and sync back to the cloud later.
• Integrated artifact generation and workflow management. Much of the development process follows pretty strict workflows that are commonly automated using continuous integration servers and build tools such as ant, rake, or maven. Most of the build tools are just trying to hide excessive amounts of pushing around of files, which we are trying to get rid off here. But the notion of automatically generating artifacts at various points in well defined work flows is of course something that is worthwhile keeping. Artifact generation could be plugin based or even similar to intentional programming style transformations.
• Distribution. When I say the cloud, I don’t mean that there should be some big monster repository of software. I’m thinking more of a REST style architecture of resources on the internet that are linked to each other. Also I like the notion of Git/Mercurial style distributed repositories of stuff. The cloud should very much be a distributed thing as opposed to some monolithic monstrous content management system.

What does all this mean in practical terms?

Having everything in the cloud like outlined above allows for the elimination of a lot of cruft and overhead in the current software engineering practice. A few examples of activities that will work in a very different way:

• Checking out code and committing changes. This is no longer needed. Since you are working with a cloud based system, all your edits go straight to the cloud. You might still want to annotate some milestones with a “commit message” of course.
• Branching and tagging. Everybody working on the same software sounds nice of course but sometimes you want to isolate changes on a branch or take a snapshot of a particular revision of the software. There’s no good reason why a cloud based system could not do this. Additionally, this removes the need for having local work copies that are ‘dirty’ instead you could just save to a private branch.
• Changes. Like in git, the cloud stores chains of incremental changes to the AST (rather than to files).
• Merging. If you have branches, there will be the need to merge changes back and forth. Since we are now all cloud based, this no longer requires pushing around vast amounts of data. Instead since all branches are just chains of changes, the process could be very similar to what git does when merging and rebasing.
• Compilation. Compiling code is the process of generating artifacts from ASTs. Since the ASTs live in the cloud and compiling is just a special case of transforming one AST to another (see intentional programming), this should pretty much be an on the fly kind of thing where the right artifacts are simply generated on the fly when they are needed. This could be an intentional programming like extensible system, which would be really nice of course. However, a plain old compiler might do the trick as well for many.
• Running tests. Running tests is an essential part of the work flow in any decent software engineering practice. Since we now have a cloud based system with explicit support for common CMS features like work flows, it is probably a good idea for the cloud to automatically run tests at the appropriate moment in a work flow (e.g. when merging from one branch to another). Some developers already use elaborate continuous integration setups involving build servers, SCM triggers, cron jobs, etc. This just takes it to the next level.
• Dependencies and relations between different systems. Software does not exist in isolation but it depends on other software. In the conventional world, dependencies need to be managed and artifacts need to be dowloaded locally. A dependency is nothing more than a special case of a relation between two nodes in an AST. The cloud is full of relations. You might know them as URLs. So, in the cloud, a dependency is just a link. Relations like this may be dependencies but there may be many types of useful relations like for example, this class node here contains tests for that class there. Links are a much more powerful construct than textual references in some file.
• Sharing software. This one is easy: you send somebody a link. That’s it, we all connect to the same cloud. The rest is just credentials, access rights, etc.
• Releasing artifacts. Since the generation of artifacts (aka. compiling) is happening on the fly, a release is nothing but the act of publishing, which in turn is a step in a workflow. Probably you will want to release related artifacts such as change logs, documentation, upgrade guides as well. These of course also live in the cloud.
• Filing bugs. Bugs are just a special case of a particular artifact that can be represented as an AST. In other words, it can be part of the cloud and be accessed and managed in the same way. There is no need for integtation of legacy bug tracking systems. Having both code and issue tracking in the same cloud of course allows for some nice workflow enhancements. Think Mylyn style workflows.
• Writing documentation. Documentation is just another example of an artifact. That too can live in the cloud and leverage the ability to have links to and from the documentation and the documented artifact(s).

Conclusion

This concludes what may seem like a pretty random collection of thoughts but is in fact something that has been brewing in my mind for about ten years in various forms.

In summary, what I’m trying to propose here is a cloud based system that has lots of fancy features like you would find in content management systems, collaborative editing tools, and can be used online and offline. For me this is not science fiction since all the bits and pieces exist already. It’s just that in the software tools used today we are stuck with nineteen seventies technology and elaborate workarounds to compensate for some of the bigger problems.

I believe that if properly implemented, this cloud based software development environment would be vastly superior to the loosely connected collection of tools, systems, and technologies that dominate the current practice. It would stimulate more effective collaboration, would vastly cut down on the tedious bureaucracy associated with software development (waiting for builds, checking out code, dealing with conflicts when synchronizing changes, etc.) and would lead to more productive development where developers are able to focus on the process of creation more than is the case now.