I was going to submit the stuff below in a shortened form as a comment to this fun little blog post on “bear shaving” but it sort of grew into a full blown article, again. To summarize the original article, there’s this nice analogy of shaving bears to help them cope with global warming and how that is not really addressing the core issues (not to mention dangerous). The analogy is applied to integration builds and people patching things up. Then the author sort of goes off and comes up with a few arguments against git and decentralization.

While some of the criticism is valid, this of course ticked me off

I see Git as a solution to increase the amount of change and dealing more effectively with people working in parallel. Yes, this puts a strain on integrating the resulting changes. But less change is the equivalent of bear shaving here. Change is good. Change is productivity. You want more productivity not less. You want to move forward as fast as you possibly can. Integration builds breaking are a symptom of a larger problem. Bear shaving would be doing everything you can to make the integration builds work again, including forcing people to sit on their hands. The typical reflex to a crisis like this in the software industry is less change, complete with the process to ensure that people do less. This is how waterfall was born. Iterative or spiral development is about the same thing but doing it more frequently and less longer. This was generally seen as an improvement. But you are still going to sit on your hands for pro longed periods of time. The real deal these days is continuous deployment and you can’t do this if you are sitting on your hands.

Breaking integration builds have a cause: the people making the changes are piling mistake on mistake and keep bear shaving (I love the metaphor) the problem because they are under a pressure to release and deliver functionality. All a faster pace of development does is make this more obvious. Along with the increased amount of change per time-unit comes also an increased amount of mistakes per time unit. Every quick fix and every misguided commit makes the system as a whole a little less stable. That’s why the waterfall model includes a feature freeze (aka. integration) where no changes are allowed because the system would never get finished otherwise.

A long time ago I wrote an article about design erosion. It was one of the corner stones of my phd thesis (check my publication page if you are interested). In a nutshell: changes are cumulative and we take design decisions in the context of our expectations of the future. Only problem: nobody can predict the future accurately and as a consequence, there will be mistakes from time to time. It is inevitable that you will get it wrong sometimes and won’t realize right away. You can’t just rip out a single change you made months/years ago without the depending subsequent changes being affected. In other words, change is cumulative: rip one piece out and the whole sand castle collapses. Some of the decisions will be wrong or will have to be reconsidered at some point and because changes are inter dependent, fixing design erosion can be painful and expensive. Consequently, it is inevitable that all software designs erode over time as inevitably such change is delayed until the last possible moment. Design erosion is a serious problem. You can’t just fix a badly eroded system that you had for years over-night. Failing to address design erosion in time can actually kill your company, product or project. But you can delay the inevitable by dealing with the problems closer to where they originate instead of dealing with it later. Dealing with the problem close to where it originates means less subsequent changes are affected, meaning that you minimize the cost of fixing the problem. Breaking integration builds are a symptom of an eroding design. Delaying the fix makes it worse.

So, the solution is to refactor and rethink the broken parts of the system to be more robust, easier to test, more flexible to meet the requirements, etc. Easier said then done of course. However, Git is a revolutionary enabler here: you can do the more disruptive stuff on a git branch and merge it back in when it is ready instead of when you go home and break the nightly build. This way you can do big changes without destabilizing your source tree. Of course you want continuous integration on your branches too. That way, you will push less mistakes between branches, thus solving problems closer to their origin and without affecting each other. You will still have breaking builds, but they will be cheaper to fix. Decentralization is the solution here and not the problem as is suggested in the blog post I linked above!

Here’s why decentralization works: testing effort grows exponentially to the amount of change. Double the amount of change, and you quadruple the testing effort. So don’t do that and keep the testing effort low. In a centralized world you do this through feature freeze. By stopping all change, you can actually find all the problems you introduced. In a decentralized world you do this by not pushing your changes until the changes you pull are no longer breaking your local branch. Then you push your working code. Why is this better? 1) you integrate incoming changes with your changes instead of the other way around. 2) you do this continuously (every time you pull changes), so you fix problems when they happen. 3) your changes only get pushed when they are stable which means that other people have less work with #1 and #2 on their side. 4) By keeping changes isolated from each other, you make it easier to test them. Once tested, the changes are a lot easier to integrate.

Continuous integration can help here but not if you only do it on the production branch: you need to do it all over the place. Serializing all the change through 1 integration environment turns it into a bottleneck: your version system may be decentralized but if your integration process is not you are still going to be in trouble. A centralized build system works ok with a centralized version system because centralized version system serializes the changes anyway (which is a problem and not something to keep bear shaving). The whole point of decentralizing version management is decentralizing change. You need to decentralize the integration process as well.

In a nutshell, this is how the linux kernel handles thousands of kloc of changes per day with hundreds of developers. And, yes, it is no coincidence that those guys came up with git. The linux kernel deals with design erosion by a continuous re development. The change is not additive, people are literally making changes all over the linux source tree, all the time. There is no way in hell they could deal with this in a centralized version management type environment. As far as I know, the linux kernel has no automated continuous integration. But they do have thousands of developers running all sorts of developer builds and reporting bugs against them, which is really the next best thing. Nothing gets in the mainline kernel without this taking place.

A bit longer post today. My previous blog post set me off pondering on a couple of things that I have been pondering on before that sort of fit nicely together in a potential way forward. In this previous post and also this post, I spent a lot of words criticizing maven. People would be right to criticize me for blaming maven. However, that would be the wrong way to take my criticism. There’s nothing wrong with maven, it just annoys the hell out of me that it is needed and that I need to spend so much time waiting for it. In my view, maven is a symptom of a much bigger underlying problem: the java server side world (or rather the entire solution space for pretty much all forms of development) is bloated with tools, frameworks, application servers, and other stuff designed to address tiny problems with each other. Together, they sort of work but it isn’t pretty. What if we’d wipe all of that away, very much like the Sun people did when they designed Java 20 years ago? What would be different? What would be the same? I cannot of course see this topic separately from my previous career as a software engineering researcher. In my view there have been a lot of ongoing developments in the past 20 years that are now converging and morphing into something that could radically improve over the existing state of the art. However, I’m not aware of any specific projects taking on this issue in full even though a lot of people are working on parts of the solution. What follows is essentially my thoughts on a lot of topics centered around taking Java (the platform, not necessarily the language) as a base level and exploring how I would like to see the platform morph into something worthy of the past 40 years of research and practice.

Architecture

Lets start with the architecture level. Java packages were a mistake, which is now widely acknowledged. .Net namespaces are arguably better and OSGi bundles with explicit required and provided APIs as well as API versioning are better still. To scale software into the cloud where it must coexist with other software, including different (or identical) versions of itself, we need to get a grip on architecture.

The subject has been studied extensively (see here fore a nice survey of some description languages) and I see OSGi as the most successful implementation to date that preserves important features that most other development platforms currently lack, omit, or half improvise. The main issue with OSGi is that it layers stuff on top of Java but is not really a part of it. Hence you end up with a mix of manifest files that go into jar files; annotations that go into your source code; and cruft in the form of framework extensions to hook everything up, complete with duplicate functionality for logging, publish subscribe patterns, and even web service frameworks. The OSGi people are moving away towards a more declarative approach. Bring this to its ultimate conclusion and you end up with language level support for basically all that OSGi is trying to do. So, explicit provided and required APIs, API versioning, events, dynamic loading/unloading, isolation.

A nice feature of Java that OSGi relies on is the class loader. When used properly, it allows you to create a class loader, let it load classes, execute the functionality, and then destroy the class loader and all the stuff it loaded which is then garbage collected. This is nice for both dynamic loading and unloading of functionality as well as isolating functionality (for security and stability reasons). OSGi heavily depends on this feature and many application servers try to use this. However, the mechanisms used are not exactly bullet proof and there exist enormous problems with e.g. memory leaking which causes engineers to be very conservative with relying on these mechanisms in a live environment.

More recently, people have started to use dependency injection where the need for something is expressed in the code (e.g. with an annotation) or externally in some configuration file). Then at run time a dependency injecting container tries to fulfill the dependencies by creating the right objects and injecting dependencies. Dependency injection improves testability and modularization enormously.

A feature in maven that people seem to like is its way of dealing with dependencies. You express what you need in the pom file and maven fetches the needed stuff from a repository. The maven, osgi, & spring combo, is about to happen. When it does, you’ll be specifying dependencies in four different places: java imports; annotations, the pom file, and the osgi manifest. But still, I think the combined feature set is worth having.

Language

Twenty years ago, Java was a pretty minimalistic language that took basically the best of 20 years (before that) of OO languages and kept a useful subset. Inevitably, lots got discarded or not considered at all. Some mistakes were made, and the language over time absorbed some less than perfect versions of the stuff that didn’t make it. So, Java has no language support for properties, this was sort of added on by the setter/getter convention introduced in JavaBeans. It has inner classes instead of closures and lambda functions. It has no pure generics (parametrizable types) but some complicated syntactic sugar that gets compiled to non generic code. The initial concurrent programming concepts in the language were complex, broken, and dangerous to use. Subsequent versions tweaked the semantics and added some useful things like the java concurrent package. The language is overly verbose and 20 years after the fact there is now quite a bit of competition from languages that basically don’t suffer from all this. The good news is that most of those have implementations on top of the JVM. Lets not let this degenerate into a language war but clearly the language needs a proper upgrade. IMHO scala could be a good direction but it too has already some compromise embedded and lacks support for the architectural features discussed above. Message passing and functional programming concepts are now seen as important features for scalability. These are tedious at best in Java and Scala supports these well while simultaneously providing a much more concise syntax. Lets just say a replacement of the Java language is overdue. But on the other hand it would be wrong to pick any language as the language. Both .Net and the JVM are routinely used as generic runtimes for all sorts of languages. There’s also the LLVM project, which is a compiler tool chain that includes dynamic compilation in a vm as an option for basically anything GCC can compile.

Artifacts should be transient

So we now have a hypothetical language, with support for all of the above. Lets not linger on the details and move on to deployment and run time. Basically the word compile comes from the early days of computing when people had to punch holes into cards and than compile those into stacks and hand feed them to big, noisy machines. In other words, compilation is a tedious & necessary evil. Java popularized the notion of just in time compilation and partial, dynamic compilation. The main difference here is that just in time compilation merely moves the compilation step to the moment the class is loaded whereas dynamic compilation goes a few steps further and takes into account run-time context to decide if and how to compile. IDEs tend to compile on the fly while you edit. So why, bother with compilation after you finish editing and before you need to load your classes? There is no real technical reason to compile ahead of time beyond the minor one time effort that might affect start up. You might want the option to do this but it should not default to doing it.

So, for most applications, the notion of generating binary artifacts before they are needed is redundant. If nothing needs to be generated, nothing needs to be copied/moved either. This is true for both compiled or interpreted and interpreted languages. A modern Java system basically uses some binary intermediate format that is generated before run-time. That too is redundant. If you have dynamic compilation, you can just take the source code and execute it (while generating any needed artifacts for that on the fly). You can still do in IDE compilation for validation and static analysis purposes. The distinction between interpreted and static compiled languages has become outdated and as scripting languages show, not having to juggle binary artifacts simplifies life quite a bit. In other words, development artifacts (other than the source code) are transient and with the transformation from code to running code automated and happening at run time, they should no longer be a consideration.

That means no more build tools.

Without the need to transform artifacts ahead of run-time, the need for tools doing and orchestrating this also changes. Much of what maven does is basically generating, copying, packaging, gathering, etc. artifacts. An artifact in maven is just a euphemism for a file. Doing this is actually pretty stupid work. With all of those artifacts redundant, why keep maven around at all? The answer to that is of course testing and continuous integration as well as application life cycle management and other good practices (like generating documentation). Except, lots of other different tools are involved with that as well. Your IDE is where you’d ideally review problems and issues. Something like Hudson playing together with your version management tooling is where you’d expect continuous integration to take place and application life cycle management is something that is part of your deployment environment. Architectural features of the language and run-time combined with good built in application and component life cycle removes much of the need of external tooling to support all this and improves interoperability.

Source files need to go as well

Visual age and smalltalk pioneered the notion of non file based program storage where you modify the artifacts in some kind of DB. Intentional programming research basically is about the notion that programs are essentially just interpretations of more abstract things that get transformed (just in time) to executable code or into different views (editable in some cases). Martin Fowler has long been advocating IP and what he refers to as the language workbench. In a nut shell, if you stop thinking of development as editing a text file and start thinking of it as manipulating abstract syntax trees with a variety of tools (e.g. rename refactoring), you sort of get what IP and language workbenches are about. Incidentally, concepts such as APIs, API versions, provided & required interfaces are quite easily implemented in a language workbench like environment.

Storage, versioning, access control, collaborative editing, etc.

Once you stop thinking in terms of files, you can start thinking about other useful features (beyond tree transformations), like versioning or collaborative editing for example. There have been some recent advances in software engineering that I see as key enablers here. Number 1 is that version management systems are becoming decentralized, replicated databases. You don’t check out from git, you clone the repository and push back any changes you make. What if your IDE were working straight into your (cloned) repository? Then deployment becomes just a controlled sequence of replicating your local changes somewhere else (either push based, pull based, or combinations of that. A problem with this is of course that version management systems are still about manipulating text files. So they sort of require you to serialize your rich syntax trees to text and you need tools to unserialize them in your IDE again. So, text files are just another artifact that needs to be discarded.

This brings me to another recent advance: couchdb. Couchdb is one of the non relational databases currently experiencing lots of (well deserved) attention. It doesn’t store tables, it stores structured documents. Trees in other words. Just what we need. It has some nice properties built in, one of which is replication. Its built from the ground up to replicate all over the globe. The grand vision behind couchdb is a cloud of all sorts of data where stuff just replicates to the place it is needed. To accomplish this, it builds on REST, map reduce, and a couple of other cool technology. The point is, couchdb already implements most of what we need. Building a git like revision control system for versioning arbitrary trees or collections of trees on top can’t be that challenging.

Imagine the following sequence of events. Developer A modifies his program. Developer B working on the same part of the software sees the changes (real time of course) and adds some more. Once both are happy they mark the associated task as done. Somewhere on the other side of the planet a test server locally replicates the changes related to the task and finds everything is OK. Eventually the change and other changes are tagged off as a new stable release. A user accesses the application on his phone and at the first opportunity (i.e. connected), the changes are replicated to his local database. End to end the word file or artifact appears nowhere. Also note that the bare minimum of data is transmitted: this is as efficient as it is ever going to get.

Conclusions

Anyway, just some reflections on where we are and where we need to go. Java did a lot of pioneering work in a lot of different domains but it is time to move on from the way our grand fathers operated computers (well, mine won’t touch a computer if he can avoid it but that’s a different story). Most people selling silver bullets in the form of maven, ruby, continuous integration, etc. are stuck in the current thinking. These are great tools but only in the context of what I see as a deeply flawed end to end system. A lot of additional cruft is under construction to support the latest cloud computing trends (which is essentially about managing a lot of files in a distributed environment). My point here is that taking a step back and rethinking things end to end might be worth the trouble. We’re so close to radically changing the way developers work here. Remove files and source code from the equation and what is left for maven to do? The only right answer here is nothing.

Why do I think this needs to happen: well, developers are currently wasting enormous amounts of time on what are essentially redundant things rather than developing software. The last few weeks were pretty bad for me, I was just handling deployment and build configuration stuff. Tedious, slow, and maven is part of this problem.

Update 26 October 2009

Just around the time I was writing this, some people decided to come up with Play, a framework + server inspired by Python Django that preserves a couple of cool features. The best feature: no application server restarts required, just hit F5. Works for Java source changes as well. Clearly, I’m not alone in viewing the Java server side world as old and bloated. Obviously it lacks a bit in functionality. But that’s easily fixed. I wonder how this combines with a decent dependency injection framework. My guess is not well, because dependency injection frameworks require a context (i.e.) state to be maintained and Play is designed to be stateless (like Django). Basically, each save potentially invalidates the context require a full reload of that as well (i.e. a server restart). Seems the play guys have identified the pain point in Java: server side state comes at a price.

I’ve had plenty of time to reflect on the state of server side Java, technology, and life in general this week. The reason for all this extra ‘quality’ time was because I was stuck in an endless loop waiting for maven to do its thing, me observing it failed in subtly different ways, tweaking some more, and hitting arrow up+enter (repeat last command) and fiddling my thumbs for two more minutes. This is about as tedious as it sounds. Never mind the actual problem, I fixed it eventually. But the key thing to remember here is that I lost a week of my life on stupid book keeping.

On to my observations:

• I have more xml in my maven pom files than I ever had with my ant build.xml files four years ago, including running unit tests, static code checkers, packaging jars & installers, etc. While maven does a lot of things when I don’t need them to happen, it seems to have an uncanny ability to not do what I want when I need it to or to first do things that are arguably redundant and time consuming.
• Maven documentation is fragmented over wikis, javadoc of diverse plugins, forum posts, etc. Google can barely make sense of it. Neither can I. Seriously, I don’t care about your particular ideology regarding elegance: just tell me how the fuck I set parameter foo on plugin bar and what its god damn default is and what other parameters I might be interested in exist.
• For something that is supposed to save me time, I sure as hell am wasting a shit load of time on making it do what I want and watching it do what it does (or not), and fixing the way it works. I had no idea compiling & packaging less than 30 .java files could be so slow.
• Few people around me dare to touch pom files. It’s like magic and I hate magicians myself. When it doesn’t work they look at me to fix it. I’ve been there before and it was called ant. Maven just moved the problem and didn’t solve a single problem I had five years ago while doing the exact same shit with ant. Nor did it make it any easier.
• Maven compiling, testing, packaging and deploying defeats the purpose of having incremental compilation and dynamic class (re)loading. It’s just insane how all this application server deployment shit throws you back right to the nineteen seventies. Edit, compile, test, integration-test, package, deploy, restart server, debug. Technically it is possible to do just edit, debug. Maven is part of the problem here, not of the solution. It actually insists on this order of things (euphemistically referred to as a life cycle) and makes you jump through hoops to get your work done in something resembling real time.
• 9 out of 10 times when maven enters the unit + integration-test phase, I did not actually modify any code. Technically, that’s just a waste of time (which my employer gets to pay for). Maven is not capable of remembering the history of what you did and what has changed since the last time you ran it so like any bureaucrat it basically does maximum damage to compensate for its ignorance.
• Life used to be simple with a source dir, an editor, a directory of jars and an incremental compiler. Back in 1997, java recompiles took me under 2 seconds on a 486, windows NT 3.51 machine with ‘only’ 32 MB, ultra edit, an IBM incremental java compiler, and a handful of 5 line batch files. Things have gotten slower, more tedious, and definitely not faster since then. It’s not like I have much more lines of code around these days. Sure, I have plenty of dependencies. But those are run-time resolved, just like in 1997, and are a non issue at compile time. However, I can’t just run my code but I have to first download the world, wrap things up in a jar or war, copy it to some other location, launch some application server, etc. before I am in a position to even see if I need to switch back to my editor to fix some minor detail.
• Your deployment environment requires you to understand the ins and outs of how where stuff needs to be deployed, what directory structures need to be there, etc. Basically if you don’t understand this, writing pom files is going to be hard. If you do understand all this, pom files won’t save you much time and will be tedious instead. You’d be able to write your own bash scripts, .bat files or ant files to achieve the same goals. Really, there’s only so many ways you can zip a directory into a .jar or .war file and copy them over from A to B.
• Maven is brittle as hell. Few people on your project will understand how to modify a pom file. So they do what they always do, which is copy paste bits and pieces that are known to more or less do what is needed elsewhere. The result is maven hell. You’ll be stuck with no longer needed dependencies, plugins that nobody has a clue about, redundant profiles for phases that are never executed, half broken profiles for stuff that is actually needed, random test failures. It’s ugly. It took me a week to sort out the stinking mess in the project I joined a month ago. I still don’t like how it works. Pom.xml is the new build.xml, nobody gives a shit about what is inside these files and people will happily copy paste fragments until things work well enough for them to move on with their lives. Change one tiny fragment and all hell will break loose because it kicks the shit out of all those wrong assumptions embedded in the pom file.

Enough whining, now on to the solutions.

• Dependency management is a good idea. However, your build file is the wrong place to express those. OSGI gets this somewhat right, except it still externalizes dependency configuration from the language. Obviously, the solution is to integrate the component model into the language: using something must somehow imply depending on something. Possibly, the specific version of what you depend on is something that you might centrally configure but beyond that: automate the shit out of it, please. Any given component or class should be self describing. Build tools should be able to figure out the dependencies without us writing them down. How hard can it be? That means some none existing language to supersede the existing ones needs to come in existence. No language I know of gets this right.
• Compilation and packaging are outdated ideas. Basically, the application server is the run-time of your code. Why doesn’t it just take your source code, derive its dependencies and runs it? Every step in between editing and running your code is a potential to introduce mistakes & bugs. Shortening the distance between editor and run-time is good. Compilation is just an optimization. Sure, it’s probably a good idea for the server to cache the results somewhere. But don’t bother us with having to spoon feed it stupid binaries in some weird zip file format. One of the reasons scripting languages are so popular is because it reduces the above mentioned cycle to edit, F5, debug. There’s no technical reason whatsoever why this would not be possible with statically compiled languages like java. Ideally, I would just tell the application server the url of the source repository, give it the necessary credentials and I would just be alt tabbing between my browser and my editor. Everything in between that is stupid work that needs to be automated away.
• The file system hasn’t evolved since the nineteen seventies. At the intellectual level, you modify a class or lambda function or whatever and that changes some behavior in your program, which you then verify. That’s the conceptual level. In practice you have to worry about how code gets translated into binary (or asciii) blobs on the file system, how to transfer those blobs to some repository (svn, git, whatever), then how to transfer them from wherever they are to wherever they need to be, and how they get picked up by your run-time environment. Eh, that’s just stupid book keeping, can I please have some more modern content management here (version management, rollback, auditing, etc.)? Visual age actually got this (partially) right before it mutated into eclipse: software projects are just databases. There’s no need for software to exist as text files other than nineteen seventies based tool chains.
• Automated unit, integration and system testing are good ideas. However, squeezing them in between your run-time and your editor is just counter productive. Deploy first, test afterwards, automate & optimize everything in between to take the absolute minimum of time. Inserting automated tests between editing and manual testing is a particularly bad idea. Essentially, it just adds time to your edit debug cycle.
• XML files are just a fucking tree structures serialized in a particularly tedious way. Pom files are basically arbitrary, schema less xml tree-structures. It’s fine for machine readable data but editing it manually is just a bad idea. The less xml in my projects, the happier I get. The less I need to worry about transforming tree structures into object trees, the happier I get. In short, lets get rid of this shit. Basically the contents of my pom files is everything my programming language could not express. So we need more expressive programming languages, not entirely new ones to complement the existing ones. XML dialects are just programming languages without all of the conveniences of a proper IDE (debuggers, code completion, validation, testing, etc.).

Ultimately, maven is just a stop gap. And not even particularly good at what it does.

update 27 October 2009

Somebody produced a great study on how much time is spent on incremental builds with various build tools. This stuff backs my key argument up really well. The most startling out come:

Java developers spend 1.5 to 6.5 work weeks a year (with an average of 3.8 work weeks, or 152 hours, annually) waiting for builds, unless they are using Eclipse with compile-on-save.

I suspect that where I work, we’re close to 6.5 weeks. Oh yeah, they single out maven as the slowest option here:

It is clear from this chart that Ant and Maven take significantly more time than IDE builds. Both take about 8 minutes an hour, which corresponds to 13% of total development time. There seems to be little difference between the two, perhaps because the projects where you have to use Ant or Maven for incremental builds are large and complex.

So anyone who still doesn’t get what I’m talking about here, build tools like maven are serious time wasters. There exist tools out there that reduce this time to close to 0. I repeat, Pyhton Django = edit, F5, edit F5. No build/restart time whatsoever.

I’m rather late with this since it has been more than a month this was news. Busy, moving to Berlin, and other lame excuses. But better late than never.

You might recall a little youtube video I posted back in October of me demoing a prototype at a press event from Nokia. As promised, but slightly later than planned, my colleagues back in Helsinki actually launched (press release) the thing in a place in Helsinki called Kamppi. Kamppi is a shopping mall plus bus station in the center of Helsinki. About 100000 people pass through the building every day. Mostly commuters but also shoppers. There’s several floors with shops, restaurants, etc. It’s an ideal setting for trialing our system and my colleagues worked hard to get the shops in Kamppi on board.

By launching I mean that we opened kamppi.nokia.mobi, which is a mobile website, for the public. You can visit this with your desktop browser of course but the website is designed to be used from a mobile phone with a mobile browser. A broad range of browsers from different phone vendors is supported but for best results you need of course the latest and greatest from Nokia. You can actually use the website from anywhere in the world though admittedly it is a bit pointless to do so unless you are planning to visit Kamppi or are actually in Kamppi (or on your way to Kamppi).

The site we launched has actually less features then the stuff we demoed in October. The reason for this was a change in focus of the trial and not the technology. The trial is now focused on indoor maps, vouchers, shop pages, and ratings. We found that the other features we demoed in October were nice but also a bit confusing to users. They may be added back later on. But since I no longer work in Helsinki, it isn’t up to me. But I put a lot of work in getting this trial going. I helped build and design the software and many of the features and had lots of fun learning Python, Django, and a load of other stuff as well as re-acquainting myself with Apache Lucene, Java, and OSGI.

Some highlights of features I actually worked on/designed:

• Implicit profiles. Save vouchers in your account without actually signing in. This is a big benefit because most users will never bother to signup. This was implemented after we implemented both OpenID and Nokia Account. In the end the usability argument won. Nobody asks for your ID when you grab a paper voucher at the entrance. Why should digital vouchers be any different?
• Search. The search server behind the scenes is based on Apache lucene and has custom extensions for indoor location tags, which we use to search through shops, vouchers, ads, etc. Many of the dynamic views have one or more search queries integrated to pull useful info out of the system. And of course there’s a search box as well. The website is in Finnish, which I don’t speak, and I configured Lucene to do Finnish stemming and tokenizing.
• I didn’t build it but one of my former master thesis students, Daniel Wilms, did build the voucher subsystem for us after I sketched the design on a whiteboard. The voucher subsystem was born quite late in the process (a few months before the October demo) just because we could. We had been looking at voucher systems for some time because the use case was interesting and we decided to just go ahead and build it. We had the first working prototype implemented in about two days. The rest of the development was just tweaking the usability.

Additionally me and Jaakko Kyro have lead a team of developers to build this that was always changing, and other stuff, through nearly two years. During this time there were many internal demos, including one to our CEO, and external demos, e.g. at the Internet of Things Conference in 2008.

So, it’s really nice that this is finally out. Go and check it out!

]]> http://www.jillesvangurp.com/2009/04/16/kamppinokiamobi/feed/ 6 http://www.jillesvangurp.com/2008/08/22/songbird-beta-07/ http://www.jillesvangurp.com/2008/08/22/songbird-beta-07/#comments Fri, 22 Aug 2008 18:22:48 +0000 Jilles http://www.jillesvangurp.com/2008/08/22/songbird-beta-07/

Songbird Blog » Songbird Beta is Released!.

Having played with several milestone builds of songbird, I was keen to try this one. This is a big milestone for this music player & browser hybrid. Since I’ve blogged on this before, I will keep it short.

The good:

• New feathers (songbird lingo for UI theme) looks great. Only criticism is that it seems to be a bit of an iTunes rip off.
• Album art has landed
• Stability and memory usage is now acceptable for actually using the application
• Unlike iTunes, it actually supports the media buttons on my logitech keyboard.

The bad (or not so good since I have no big gripes):

• Still no support for the iTunes invented but highly useful compilation flag (bug 9090). This means that my well organized library is now filled with all sorts of obscure artists that I barely know but apparently have one or two songs from. iTunes sorts these into compilation corner and I use this feature to keep a nice overview of artists and complete albums.
• Despite being a media player with extension support, there appears to be no features related to sound quality. Not even an equalizer. Not even as an extension. This is a bit puzzling because this used to be a key strength of winamp, the AOL product that the songbird founders used to be involved with.
• Despite being a browser, common browser features are missing. So no bookmarks, no apparent RSS feed, no Google preconfigured in the search bar, etc. Some of these things are easily fixed with extensions.

Verdict: much closer than previous builds but still no cigar. Key issue for me is compilation flag support. Also I’d really like to see some options for affecting audio playback quality. I can see how having a browser in my media player could be useful but this is not a good browser nor a good media player yet.

]]> http://www.jillesvangurp.com/2008/08/22/songbird-beta-07/feed/ 0 http://www.jillesvangurp.com/2008/04/06/modular-windows/ http://www.jillesvangurp.com/2008/04/06/modular-windows/#comments Sun, 06 Apr 2008 08:44:25 +0000 Jilles http://www.jillesvangurp.com/?p=401

There is a nice article on Ars discussing Microsoft’s business practices regarding windows and how they appear to be not quite working lately. It used to be that your PC came with windows whereas nowadays you have to select from a around five different versions and Microsoft is rumored to go to an even more modular and subscription based model. The general idea is to be able to squeeze out as much revenue out of the market as possible. On paper it sounds good (for MS that is).

Rather than buying an overpriced OS with everything and the kitchen sink you buy what you need. There’s a huge differences between what businesses and some individuals are willing to spend and the typical home user that just wants a browser + skype + the sims. Typically the latter group ends up buying the cheapo version and the former group ends up buying the everything and the kitchen sink version. The problem is that there is unmonetized value in the latter in the sense that some owners of the  cheapo versions might be interested in getting access to some of those features in the expensive version but not in all of them.

Now to the obvious problem with the discussed solution. By selling cheapo versions with most of the value removed and factored out into separate chunks you have to pay for, you dilute the overall value of the OS. So instead of buying an OS that can do X, Y, and Z out of the box you are buying an OS that can’t do X, Y, and Z out of the box. Marketing an OS that can’t do stuff is a lot harder than trying to sell stuff that can do things.  Worse they are opening the market to third parties that might do something similar to X, Y, and Z for a better price, or in some cases for free (beer & speech). Or even worse to companies selling an alternative OS with X, Y, and Z.

That in a nutshell is what is discussed in the Ars article and why Apple Mac OS X marketshare is approaching double digit percentages. I’ve been giving it some serious thought lately and I’m also noticing the spike in Safari users in my web site statistics.

Anyway, the reason for this write up is that the article overlooks an important argument here that I believe is relevant for more markets than just operating systems. In general, the tie between OS and features such as photo galleries, online backups, or TV UIs is artificial. Microsoft only adds features like this to make the overall OS more valuable. That is, they are looking to improve the value of the OS, not the photo gallery. However, ongoing and inevitable commoditization of software actually shifts value to new features. Especially when bundled with online subscriptions, things like online photo galleries can be quite good business. For example, Flickr has many paying subscribers.

Naturally MS is interested in markets like this (which is why they are interested in Yahoo). However, the tie-in to the OS constrains the market. Why would you not want to sell these services to Apple users? Why would you not want to sell this service to Sony Playstation owners? Why would you want to want to artificially limit who can access your service just to boost sales of your OS? As long as you were trying to artificially (and apparently for MS illegally) boost value of your core OS, bundling was a valid strategy. However, as soon as your value shifts, that becomes a brake on market growth. The OS market has commoditized to the point where you can get things like Ubuntu for free, which for the low end market is about as good as what you get with the cheapo version of Vista (see my various reviews of Ubuntu for why I’m not ready to claim better yet).

So the difference between MS and Google who is eating their lunch in the services arena is that the latter is not handicapped by 20 years of Windows legacy and can freely innovate and grow marketshare without having to worry about maintaining a revenue stream from legacy software. Google doesn’t have to sell OS licenses and so they give away software on all platforms to draw more users to their services which is where they make their money.

Naturally, Google has a lot of software engineers that are working round the clock to create more value for them. Where possible Google actively collaborates with the open source community because they know that while they won’t make any money from commodities like browsers, file systems and other important software components, they do depend on those things working as good as possible and keep evolving in the right direction. Few people appreciate this but this and not ads is why Google sponsors Firefox. It’s a brilliant strategy and it is forcing their main competitor to keep investing in internet explorer rather than being able to shift resources to directly competing with Google. 50 million $ is pocket money if it is making your main competitor crap their pants and waste resources on keeping up with you in a market where you are not even trying to make money.

You might have noticed that I have carefully avoided discussing Google and Microsoft’s mobile service strategies and also noticed that yours truly is working for Nokia. Well, my readers ought to be smart enough to figure out what I’m trying to say here aren’t you ?

]]> http://www.jillesvangurp.com/2008/04/06/modular-windows/feed/ 0 http://www.jillesvangurp.com/2008/02/11/sensei/ http://www.jillesvangurp.com/2008/02/11/sensei/#comments Mon, 11 Feb 2008 11:57:20 +0000 Jilles http://blog.jillesvangurp.com/2008/02/11/sensei/

Last month a European funded research project I’m in kicked off: SENSEI. The official project website is still under construction but somebody was kind enough to blog about it at least.

Sensei will be a lot of work and hopefully also very interesting and fun. It’s a so-called integrated project in FP7 which means it is one of the many projects with a lot of partners (21 in our case) across Europe of both academic and industrial side that are financed by the EU in the context of the seventh framework programme that will distribute billions of research money over the next few years. So if you are an EU citizen, thanks very much for making this possible with your tax money .

Nokia is in Sensei for 9 person years spread over three years (50% funded). The blog post mentioned above has a pretty good overview of the key points of Sensei so I won’t repeat them here. My personal interest in this project will be on the software middleware layer.

]]> http://www.jillesvangurp.com/2008/02/11/sensei/feed/ 0 http://www.jillesvangurp.com/2007/09/24/digiboksi/ http://www.jillesvangurp.com/2007/09/24/digiboksi/#comments Mon, 24 Sep 2007 16:50:14 +0000 Jilles http://blog.jillesvangurp.com/2007/09/24/digiboksi/

Digiboksi is Finglish for DVB-C set top box. I bought one on saturday and had it replaced today by a different one.

Basically the story is that Finland is replacing good old analogue TV with digital tv. Terrestial analogue signal went dark a few months ago. A few weeks ago the ‘interesting’ channels (i.e. the ones in english) disappeared from the cable and the rest will follow in February.

So, I went to the shop for a digiboksi. Since I barely watch TV, I just wanted something that was cheap & something that worked. So I pretty much randomly selected the Samsung DCB-B263Z in the shop (hey it was black, matches my other equipment). Normally I’d do some research on the internet for such a purchase. Unfortunately this product seems to be specialized for the Finnish market so most info available seems in Finnish. Hence the randomness.

So I bought it, plugged it in and about ten minutes later was watching TV. I was quite pleased with the picture quality. The UI seemed nice too. Then it crashed. WTF! Anyway, to keep this short: it is a known issue, basically the shops are selling products with broken software. Sigh. So, after about 4 crashes in less than 24 hours, I went back to the shop today and mentioned the word crash. No need for further explanation. Apparently, lots of people are bringing these things back (I know at least one other guy). Five minutes later I walked out with a Handan 3400.

So far it seems reasonably well behaved though the picture quality is slightly less nice than the samsung (was a bit smoother and crispier). I managed to improve it slightly by disabling the built in contrast. If this one crashes as well, I’ll try another brand.

]]> http://www.jillesvangurp.com/2007/09/24/digiboksi/feed/ 3 http://www.jillesvangurp.com/2007/07/04/links-for-2007-07-04/ http://www.jillesvangurp.com/2007/07/04/links-for-2007-07-04/#comments Wed, 04 Jul 2007 07:18:57 +0000 Jilles http://blog.jillesvangurp.com/2007/07/04/links-for-2007-07-04/
• Cambridge in Colour – Digital Photography Tutorials
  Great website on photography. Mandatory reading for any digital photographer who wants to improve his skills.
  (tags: photography photoshop tutorial)
• Digital Photography School
  Great website where digital camera enthousiasts exchange tips and advice.
  (tags: photography photoshop)
]]> http://www.jillesvangurp.com/2007/07/04/links-for-2007-07-04/feed/ 0 http://www.jillesvangurp.com/2007/07/03/new-photos-added/ http://www.jillesvangurp.com/2007/07/03/new-photos-added/#comments Tue, 03 Jul 2007 19:09:49 +0000 Jilles http://blog.jillesvangurp.com/2007/07/03/new-photos-added/

A few weeks ago I was on vacation in Barcelona & Catalunya (and also France for a few days). I took lots of photos and have finally found the time to properly edit them. The result is available on my photo site, as usual.

I’ve learned quite a bit about photography from a number of web sites with excellent, in depth information on a wide range of photoshop and photography related subjects that I’ve tried to apply in practice. Quite many of my photos make use of contrast masks as well as overlay blended layers with white and black spots to brighten and darken bits of the photos. I fixed some difficult light conditions and also some crappy photos where I messed up aperture or shutter settings on the camera. Some of the photos I’m still not happy with but quite many turned out quite nice.

As ever the panorama’s were produced with hugin. This great tool is much easier to use, though still not for the faint of heart.

Here’s an overview of some sites related to photography that I have found extremely useful and informative:

• www.cambridgeincolour.com – This web site is worth spending some time on, especially if you own a decent camera.
• www.luminous-landscape.com – This site offers lots of tutorials of varying quality. I found some of the material in the understanding series quite informative.
• digital-photography-school.com – This is a site with lots of tutorials and opinions on digital photography.
• My del.icio.us links related to photography – I’m trying to keep this up to date as I find more sites.
]]> http://www.jillesvangurp.com/2007/07/03/new-photos-added/feed/ 0 http://www.jillesvangurp.com/2007/05/20/links-for-2007-05-20/ http://www.jillesvangurp.com/2007/05/20/links-for-2007-05-20/#comments Sun, 20 May 2007 07:19:01 +0000 Jilles http://blog.jillesvangurp.com/2007/05/20/links-for-2007-05-20/
• 5 Techniques for Enhancing Contrast in Digital Photos
  Useful advice on improving contrast. Three out of five discussed techniques were new to me and quite useful addition to my arsenal of techniques.
  (tags: photoshop photography tutorial)
]]> http://www.jillesvangurp.com/2007/05/20/links-for-2007-05-20/feed/ 0 http://www.jillesvangurp.com/2007/05/07/links-for-2007-05-07/ http://www.jillesvangurp.com/2007/05/07/links-for-2007-05-07/#comments Mon, 07 May 2007 07:19:00 +0000 Jilles http://blog.jillesvangurp.com/2007/05/07/links-for-2007-05-07/
• Realistic photo print – Easy to follow photoshop tutorials
  useful little photoshop tutorial for making fake photo prints
  (tags: photoshop webdesign)
• Photoshop LAB color: Chapter 14: “Once for Color, Once for Contrast” – Digital Grin Photography Forum
  Great way to boost contrast
  (tags: photoshop)
]]> http://www.jillesvangurp.com/2007/05/07/links-for-2007-05-07/feed/ 0 http://www.jillesvangurp.com/2007/02/09/skype-logitech-usb-headset-350/ http://www.jillesvangurp.com/2007/02/09/skype-logitech-usb-headset-350/#comments Fri, 09 Feb 2007 18:22:36 +0000 Jilles http://blog.jillesvangurp.com/2007/02/09/skype-logitech-usb-headset-350/

After years of delaying, I’ve finally bought a headset and installed Skype. My new screenname is jillesvangurp.

The headset is a Logitech USB Headset 350. It’s a nice plug and play USB thingy that I can use next to my normal audigy 4 sound card. I.e. I can use my fine Hennheiser HD 280 Pro  for gaming, music and movies and answer the phone on my headset. I like it that way, the noise damping effect of the Sennheiser would just be weird when hearing yourself talk.

The headset installed fine in true plug and play fashion. It was recognized right away by windows as a USB headset. It’s a nice thingy with a volume control, a big mute button and a nice blue led blinking all the time.  The sound is good too and it feels quite comfortable.

Skype 3.0 is a nice piece of software as well. I had no trouble setting up a call to my friend Christian Del Rosso who has a webcam as well as a micropone. Well almost no trouble, he had some issues with the volume and it took some fiddling to figure out that in fact there was no problem with my headset.

]]> http://www.jillesvangurp.com/2007/02/09/skype-logitech-usb-headset-350/feed/ 0 http://www.jillesvangurp.com/2006/10/17/photography/ http://www.jillesvangurp.com/2006/10/17/photography/#comments Tue, 17 Oct 2006 20:19:27 +0000 Jilles http://blog.jillesvangurp.com/2006/10/17/photography/

Like many geeks, I’ve been an enthusiastic digital camera owner for some years now who, so far, is not troubled too much by in-depth knowledge of camera’s, lenses and what not. Basically I fiddle with the controls until it looks ok on the screen. Typically I assume a pose that is not really elegant (camera at arms length, chin on the chest and frowning at the minuscule screen). And then I point, click and hope for the best. Sometimes this produces nice results but producing nice results when you want them under difficult light conditions (too much or too little) is quite a challenge.
I was editing my Paris photos and did some google searches to figure out how to make more effective use of photoshop. It turns out that the new Google search field in Firefox 2 has gotten a lot smarter. I typed “photos” and google already provided a nice autocomplete list with highly relevant options such as “photoshop curves”, which led me straight to this more than excellent site which I have been reading the entire evening: http://www.cambridgeincolour.com/. The site provides a nice technical introduction to various important concepts such as depth of field, focal length & aperture, various types of histograms and how to interpret them, the working of the level and curves tools in photoshop, etc. The site is organized around tutorials and provides great examples that make it easy to understand the concepts under discussion. I’m now planning to work my way through all of the site.

]]> http://www.jillesvangurp.com/2006/10/17/photography/feed/ 0 http://www.jillesvangurp.com/2006/03/30/canon-s80/ http://www.jillesvangurp.com/2006/03/30/canon-s80/#comments Thu, 30 Mar 2006 17:58:58 +0000 Jilles http://blog.jillesvangurp.com/2006/03/30/canon-s80/

Today I shot a few pictures with my new camera. Wow. It’s very nice. It performs pretty much as explained on dpreview. Conditions were not ideal for photography. It was very cloudy and rainy. There was a thick layer of clouds and the landscape was still covered in (melting) snow. This is difficult for the camera because it has to compensate for the snow and the lack of light. Overall, the camera performed quite well.

As predicted, the camera suffers from quite a bit of color noise if iso values are increased. Basically, anything above 100 has noise. At iso 400, the noise is so bad that this setting should be avoided most of the time. Essentially this is an issue with all digital compact cameras. Other than this, image quality is excellent. And of course the image resolution is 8MP, so a bit of noise is not that bad.

Besides, filtering away color noise is something photoshop is much better at than most cameras. One of the things that attracted me to this camera is that it doesn’t overdo it on filtering noise and throwing away all the details in the process. Similarly, it doesn’t ‘improve’ the contrast which also translates into more detail. The downside of course is that if you want this, you need to tune the pictures yourself in photoshop. That is no problem for me.
Lets start with the lens. This is a 28mm equivalent camera which means that it has a slightly wider angle of view than normal 35 mm equivalent cameras. This is very nice since that means you don’t have to walk all the way back to get your subject framed. It also has a nice zoom range (3.6x). If find that shots at 3x zoom both with this and my previous camera to be quite unmanagable unless the camera is on some stable surface. The resolution of the pictures is of course excellent so I can crop away large portions of the photo and still have something that looks great when printed.



The picture above, is a good example of a wide angle shot. Sadly I had to throw away a lot of detail to get the picture down to a web friendly size. Also the picture was touched up in photoshop a bit to cover up for the fact that this was not really a good shot (sun is behind the building, well the clouds).



This shot is probably the best of the shots I made today. Basically, I walk this path every morning on my way to work. To the right (of the path) is the (frozen) sea. The only things I manipulated in this picture are the levels and curves. Because of the poor light conditions, the original is a bit dark. Also, I must learn to keep an eye on the histogram. Probably I could have gotten a bit more out of the situation with a bit more fiddling with the camera.
To give an impression of the number of pixels that were in the original of this picture, here’s a cropped piece of the same shot:


So, quite impressive amount of detail. And again, the conditions were really bad for taking photos. This shot was taken at 1/60 second shutter speed. You can’t go much lower without risking a lot of motion blur.

This camera is really easy to use and has a nice user interface. Two elements that not all reviewers enjoyed are the sliding cover (which doubles as on/off switch) and the rotating click wheel thingy (sort of a mechanical equivalent of the ipod wheel). I must say I like both. Especially the wheel thingy makes it really easy to adjust things like shutter speed, aperture or even focus. Much nicer than messing with tiny left & right buttons (as on my powershot A40).

You can find the shots above, and some more, on my photo site.

]]> http://www.jillesvangurp.com/2006/03/30/canon-s80/feed/ 0