Like most in our industry I’ve been exposed to agile, scrum and related hypes over the past decade. Unlike most in our industry, I have a little bit of background in software engineering research as well. During my Ph. D., which was in the early days of agile when Kent Beck published his book on extreme programming and I was writing my thesis on software architecture, I had the privilege of seeing a lot of different companies from the inside and talked to tech leads, architects and other leaders across a nice range of companies doing software product development. Ultimately after wrapping up my thesis, I decided that I lacked the experience to speak authoritatively on the subject of software development and moved over to the software industry to get some experience.

Continue reading →

There is a bit of a debate raging in the last few weeks that started when somebody commented on a few things that Martin Fowler wrote about git and using feature branches and feature toggles. Martin Fowler makes a few very good points about how feature branches can cause a lot of testing and integration headaches and lot of people seemed to have picked up on this one and there seems to be a bit of an anti-feature branch movement emerging.

The main problem I have with this is that these people are confusing problems and causes here and effectively blaming a solution they have for causing a problem they have. Roughly the argument goes as follows: feature branches cause people to accumulate change that then becomes very disruptive when it lands on the main branch. There will be CI breakage and lots of problems for people on other feature branches that are suddenly faced with merge issues. I think Martin Fowler actually gets it but some of his followers seem to be confused. By all means, feature toggles are a great way to get changes in early and have them exposed. It’s a powerful tool that you can use to get changes out earlier. Early is good: use it. However, it is not the case that if you use feature branches there has to be a lot of pain for everybody. In other words, feature branches are not inherently evil and don’t have to be problematic.

It’s not the practice of feature branching that is the problem but the fact that testing and continuous integration are not decentralized in a lot of organizations. In other words until your changes land on the central branch, you are not doing the due diligence of testing. Even worse, you are not making sure you have tested your changes before you add them to the main branch.

You can’t do decentralized versioning unless you also decentralize your testing and integration. Git has value when used as a SVN replacement. Git has more value when used as a DVCS. There is no good reason why you can’t do decentralized testing and integration with git. Rather the opposite: it has been designed with exactly this in mind. The whole point of git is divide and conquer. Break the changes up: decentralize the testing and integration work and solve the vast majority of problems before change is pushed upstream. If you push your problems along with your changes you are doing it wrong. Decentralized integration is a very clever strategy that is based on the notion that the effort involved with testing and integration scales exponentially rather than linearly with the amount of change. By decentralizing you have many people working on smaller sets of changes that are much easier to deal with: the collaborative effort on testing decreases and when it all comes together you have a much smaller set of problems to deal with.

This is how the linux kernel manages to remain stable, despite the fact that the amount of change there is measured in thousands of LOC/day. If you need proof that thousands of people can work collaboratively on millions of lines of code using git branches: look at the linux kernel. The amount of change, integration and testing effort, etc. in whatever you are working on probably pales in comparison with that: your problems are easy.

Here’s a few simple practices that will address most of the issues:

1. No change that will break CI on the main branch is allowed on the main branch. Zero tolerance on this one. In git terms: rebase against main, run ci test on the feature branch, fix any problems, push. You can automate this even: jenkins pretty much supports this out of the box. If main breaks ever, somebody doesn’t get the basics: educate them with a big clue bat. Rationale, it is vital to keep main stable at all times. That way everybody on a feature branch will know it is safe to rebase.
2. Rebase against main frequently, especially if you do big changes. Rationale: you are doing the changes, it is you that will take the pain of doing the integration work when things go wrong, not everybody else. The earlier you know about problems, the easier it is to fix. Feature branches are not about stopping rebases. If your feature branch is way behind, you have done something very wrong. Don’t ever do that without good reason. If it is on the central branch, you will have to deal with it at some point: so get it over with ASAP.
3. Commit frequently, keep commits as small as you can. Rationale: smaller commits are easier to analyze and fix when you have conflicts. Also, Git is really good at applying commits one by one. If you isolate the merge problems to a handful of commits, rebasing is pretty much painless.
4. Push as early as you can. If the CI builds are green on your branch and you are confident things are fine, push and don’t wait. Don’t accumulate integration work for others. Feature branches are not about hiding change but about isolating change. There’s a difference. One is a communication problem and the other is a proven strategy of divide and conquer. If appropriate, feature toggles are indeed a great way to land experimental changes. I believe this was the main point Martin Fowler was trying to make.
5. Communicate clearly around big code restructurings. Rationale, everybody rebasing against your changes will experience some pain. You are causing people to have to do work, so tell them it is going to happen. I always ask people to push their changes before I push my big changes. That way, I can fix the integration issues on my side before I push.
6. Collaborate with people by pushing changes back and forth without involving the main branch. Git format-patch is your friend: you can do this by email. If somebody needs a change before you land it, you can give it to them.
7. Be aware of the cost of things. Any time people spend on things like resolving merge conflicts, doing rebases, etc. costs you. Inevitably when you branch you accept that there is going to be some cost. Keeping a branch alive means you add cost. Don’t do it needlessly. So, branch and do what you have to do and then rebase, push and delete the branch. And again, not committing is effectively the same as branching in git. It has the same cost and risk attached to it.
8. Don’t push branches to origin unless they are going to be long lived and need to be worked on by multiple people. For simple work, keep the branches local to your machine. If you are going to be doing all of the commits and integration work, the only valid reason for pushing it upstream would be backups. There are alternatives to backing this way.
9. Beyond a certain team size, the stable branch needs to be protected. Pull change rather than pushing it (this is a severely underused git feature). Junior on the third floor says his patch is ready: pull it, have a look at it and give him feedback but don’t allow him to push and bypass checks and balances. Push only works in small teams. Pull forces people to communicate.

When will (feature) branches get you in trouble? Antipatterns:

• You are working on a big change. You haven’t updated for days. Do I need to spell it out? That is just wrong. Update!
• You are pushing a big change, all the CI builds go red. Oops, test before you push you dumb idiot. You deserve all the angry looks you are going to get. If this happens a lot, consider setting up your CI environment for having a stable branch for tested commits and an unstable branch for incoming commits that need to be tested. Jenkins supports this and it will keep unstable change out of stable.
• You are doing a big change, somebody else is also doing a big change. You find out about that when you rebase and spend hours dealing with merge conflicts. Seriously: communicate before you do anything drastic and give people an opportunity to get their changes in before you ruin their day. Pushing massive changes that you know are going to cause problems when people rebase is a very egocentric thing to do. Be a team player.
• You are working on a big change. You haven’t updated or committed anything for several days on your local branch. You are effectively using your local branch as a feature branch: everybody who is not pushing change is effectively on a feature branch. Not committing is NOT a strategy to avoid using feature branches. Also, you are not committing? Why???? What’s your excuse for not committing to your own local branch? Seriously, consider using version management and stop treating git as a file server for code backups.
• You are working on a branch for an extended period of time but your CI builds only run on the central branch. Congratulations, you have just tossed out CI as a good practice. Fix it. Either have the discipline to run the CI tests manually on every commit to the branch or set up a CI build for it. Either way, don’t break the feedback loop you get with CI.

Now at this point I have to admit something: I don’t use feature branches a lot but I do tend to accumulate a few commits locally before I push them. Also. we haven’t set up stable and unstable branches in jenkins (yet, planning to). We have the occasional breakage of our CI builds. I’m actually guilty of breaking some of the builds myself. The reason/weak excuse is: I’m having a hard time changing people’s way of working. You turn your back and people stop committing and continue treating git like they are using cvs. But I’m at least aware what the real problem is here: not the fact that I have branches but the fact that our testing needs to be decentralized.

If you are a regular, you may have noticed that I switched theme. I was using the excellent fusion theme that I found on this site, which like the theme seems to be no longer maintained. So when I upgraded to wordpress 3.2 this morning, I noticed there is a new default theme in there that I kind of like.

The best feature is support for multiple header photos. So I’ve cropped and uploaded a few photos from my collection and the site will now randomly pick one of the photos listed below. So simple, but so nice to have. Otherwise the theme is pretty basic, which I think is appropriate for a blog. Also it does microformats (which was a major annoyance with the old theme). Finally, it just looks nice and it is likely to be maintained for a while (with it shipping with wordpress and all).

After increased link spam activity (somebody found a way around the captcha) I decided to migrate the commenting system on this site to disqus.com. Along with (hopefully) a more spam resistant site, also come some cool features such as Facebook and Twitter login and a few other things. Installation was surprisingly easy. Sign up, install plugin, configure, export comments, done. I should have done this ages ago.

Update 11 July 2011: I’ve done several commits on github since announcing the project here. New features have been added; bugs have been fixed; you can find jsonj on maven central now as well; etc. In short, head over to github for the latest on this.

I’ve just uploaded a weekend project to github. So, here it is, jsonj. Enjoy.

If you read my post on json a few weeks ago, you may have guessed that I’m not entirely happy with the state of json in Java relative to other languages that come with native support for json. I can’t fix that entirely but I felt I could do a better job than most other frameworks I’ve been exposed to.

So, I sat down on Sunday and started pushing this idea of just taking the Collections framework and combining that with the design of the Json classes in GSon, which I use at work currently, and throwing in some useful ideas that I’ve applied at work. The result is a nice, compact little framework that does most of what I need it to do. I will probably add a few more features to it and expand some of the existing ones. I use some static methods at work that I can probably do in a much nicer way in this framework.

Continue reading →

Click to see the slightly larger version.

Here’s a nice Panorama of Lisbon that I shot a few weeks ago using my new Canon 550D. It consists 10 individual 18 mega pixel photos at full focal length (135mm) to zoom in across the Tejo river (around 1500m) that were stitched together using the ever amazing Hugin. The latest version of this is quite amazing at basically figuring out everything by itself. It’s like here’s a bunch of pictures, make a panorama out of that. And then it goes off and finds common points in the photos, arranges them in a grid that makes sense. Adjusts for lens characteristics, rotates, translates and distorts the photos until they overlap perfectly, adjusts tonality of the photos to match, blends the photos seamlessly and finally spits out a big whopping tif file. There are other tools that do some of these things but this one is the best by far. If you are willing to tweak the tool, it can do some pretty funky things like HDR panoramas (i.e. stitch together overlapping photos with multiple exposures), do panoramas where the photos have different exposures and tonality, use different projections for the panorama, drag around the stitched panorama inside whichever projection you chose to influence how things look, manually identify overlapping points, plugin experimental tools and algorithms for the various steps of creating a panorama, etc.

Unfortunately, you have to down-size this kind of images for the web and apply lots of jpeg compression. The final image is 10032×1024 pixels, downsized from 21896×2235 to save some precious bandwidth. The tif weighs in at a bit over 100MB. The lowests size jpeg I managed to get out of the full image was around 2.5 MB, but at that size compression artifacts are very noticeable. So I instead re-sized to something that might fit on an over sized display (vertically at least) so that people will have to scroll in only one dimension and got the size down to 1.1 MB with a respectable compression rate of around 30%, which preserves a lot of detail.

I was positioned in front of the big Jesus statue that is visible from Lisbon. I later went up as well and shot some more photos, some of which you can admire here (along with the rest of my photos from that trip). There is one photo there with the same view in a single photo from up on the statue that gives you a nice impression of the zoom range of this camera. It looks like a panorama photo but it is really just a zoomed out photo that I cropped a little.

I had a captcha plugin that was failing. Since I myself hate captcha’s, I’ve removed it for now and adjusted the comment settings to require moderation the first time you comment. Moderation + akismet should take care of most of the spam (hopefully). You can also still sign in using OpenId if you prefer, although the openid login on wordpress remains quite underwhelming (compared to what should be possible by now).

Anyway, happy commenting and thanks @eelkefolmer for pointing out my site was broken.

Time for a long overdue Saturday morning blog post.

Lately, I’ve been involved in some interesting new development at work that is all about key value stores, Json, Solr and a few other technologies I might have mentioned a few times. In other words, I’m having loads of fun at work doing stuff I really like. We’ve effectively tossed out XML and SQL and are now all about Json blobs that live in KVs and are indexed in Solr. Nothing revolutionary, except it is if you have been exposed to the wonderfully bloated and complicated world of Java Enterprise Edition and associated technologies. I’ve found that there is a bunch of knee-jerk reflexes that causes Java developers to be biased to be doing all the wrong things here. This is symptomatic of the growing gap between the enterprise Java people on one side and the cool hip kids wielding cool new languages and tools on the other hand.

Continue reading →

I just visited the Adobe online store and eventually left in disgust because apparently it is unable to service me. This is not the first time something like this has happened to me so I decided to spell it out a little.

Once upon a time, I signed up on the Adobe site to use the online version of photoshop. I probably was still in Finland at the time so naturally, I used their English localized web site. Fast forward to today and I am interested in downloading the trial version of Adobe Lightroom to play with it and possibly buy it.

To do that you need to log in. So I log in. Then it asks me for my address. Fine. Except, I don’t live in the US and there is no country switching thingy in the form. Apparently it hardwired the country choice (which really was a language preference) into my account. Fail #1.

It gets worse. They do have a country switcher on the front page and in the store. So I set that to Germany. Now it addresses me in German. Fail #2

Of course there is no language switcher. It is assumed (wrongly) that I must be a native speaker and therefore want to be addressed in German. Always. Fail #3

So by this point I decide to be pragmatic and select some region they haven’t localized for to get the site in English. So I select Eastern Europe, which is apparently a region not worth breaking up into countries and technically I live in the former DDR, which used to be eastern Europe. So I once again go to the store. Prices in euros this time. Great! Although effectively the price just went up 80 euros since 300$ is ~220 Euro and not 300 Euro! Then I get to the point I need to enter my address, it still insists on a US address. Fail #4. If people select a region, it’s probably because they live there.

Underlying this problem is a common failure of online retailers to grasp the basics of internationalization and demographics that is causing them to lose the business of millions of potential customers who have a language/region combination that they can’t support. This is especially silly when the language is English and the online retailer is US based. English is a great language. World wide, there must be many more non native speakers of it than native speakers. Especially migrants (like) me tend to rely heavily on English.

So, I am not going to be an Adobe customer.

I’m not an iTunes user either for the same reason and for the reason that I’m not big into German dubbed movies and schlagers, which is pretty much what they tried to sell me on the few times I tried to use it. Also, I declined to accept the terms of use when I was living in Finland because they were in Finnish, which is something I don’t speak and I tend not to agree with official looking text that I can’t read.

I’ve had the nokia N8 for a few weeks now and since I like it, I thought I’d do a little review.

Does it have rough edges? Sadly, yes there are some. But no major show stoppers. It’s stable. The camera is awesome. The video playback is great. You can play most movies without re-encoding them. I managed to not run out of battery during a trans atlantic flight, watching movies non stop, and the oled screen is fantastic.

I’m particularly happy with how maps is improving with each version.

It went from something that barely worked a few years ago to being arguably one of the best mobile maps experiences around. It’s certainly very competitive when it comes to map quality, navigation, and offline usage.Disclaimer, I work on the backend services for this application .

I’ve used two ovi store applications to write this post. One is called swype and it replaces the touchscreen keyboard with a version that allows you to ‘draw’ across the keyboard and it figures out the words with great accuracy and speed.

The other is the official wordpress client, which I’m trying for the first time, and of course that’s the real reason for writing all this .

So far so good. I managed to write a good chunk of text with swype and the wordpress app seems to do everything I need it to do.

All together this is a pretty good phone and it is getting even better with upcoming software releases. I think the recent focus of nokia on software quality and end user experience is starting to pay off. It’s not without flaws of course and it’s definitely to early to call the battle with our competitors over but it is progress and I think we’re getting there.

A final note here. I work for Nokia and am of course slightly biased. That being said, I rarely review stuff we produce and as a rule only when I really like it, without reservations. I loved the n900, which I reviewed a few months ago, but I’ve had several nokia phones that I chose not to review as well for this reason.