Moving to Berlin

A bit more than a month ago, I posted a little something on the reorganization in Nokia Research Center where I work and announced my availability on the job market. This was a bit of a shock of course and it has been a hectic few weeks but the end result is really nice. For me at least. Unfortunately some of my colleagues are not so lucky and are now at risk of losing their job.

In any case, a few weeks ago I visited Nokia Gate5 in Berlin for a job interview. Gate5 is a navigation software company that Nokia bought in 2006. Their software is powering what is now known as OVI Maps and whereas the whole industry is shrinking, they are growing like crazy now and rolling out one cool product after another. Today, they sent me a proposal for a contract. Barring contractual details, this means that I will be based in Berlin from February. This is something I’ve known for a few weeks but having all the necessary approvals from Nokia management and a concept contract is about as good as it gets in terms of certainty. So, since I know a few people are curious what I’ll be up to next year, I decided on this little update.

I can’t say too much about what I will do there except that it more or less matches my Java server side interests and experience perfectly. This means back to being a good old Java hacker which is just fine with me and something I’ve not had enough time to focus on lately (much to my annoyance). Just today I submitted an article and I have one or two other things to finish off in January. After that, my research will be put on hold for a while. That’s fine with me as well. After returning to a research career three years ago, I’ve done a few nice papers but to be honest, I’m not enjoying this as much as I used to.

Of course Berlin is a great place to move to. I’ve been there now twice. I remember thinking the first time in 2005 that “hmm, I wouldn’t mind living here” and when I was there three weeks ago I had the same feeling again. It’s a big city with a rich history, nice culture and lots of stuff to see and do. I also learned that this is one of the few cities in Europe where life is actually cheap. Apartment prices, food, drink and all the essentials in life are really affordable there and of excellent quality too.

Anyway, I’ll be off to France the next week visiting my parents.

Happy holidays

server side osgi, a myth?

Two years ago, I started using OSGI, the popular Java dependency injecting component standard, for an internal project. Fast forward to now and I have a nice set of bundles that depend on, amongst other the OSGI HTTP service.

All along, I’ve been reading how great OSGI is and how flexible it is and how it is the future of server side Java. I was ready to believe it. But to cut to the meat of this blog post: server side OSGI is vaporware. It doesn’t exist. None of the vendors actually support it. Support it as in production quality, well documented, widely used product available right now. I’ve looked at Felix, Tomcat, Equinox,  Jetty, Glassfish, JBoss, etc. and came up with nothing but a few obscure, unsupported, undocumented components. The default HTTP service implementation is not my idea of scalable & production quality. And the connections of existing production quality OSGI containers to existing production quality application servers is sketchy at best.

Frankly, I’m very surprised at this.I know lots of people that claim use OSGI serverside and there are are lots of announcements of vendor X endorsing OSGI bla bla bla fully modularized bla bla bla dependency injection  bla bla bla. That’s great but after two years of OSGI hacking I was hoping for something a little more substantial than what I have found so far:

The best option I came up with is the HTTP servlet bridge from equinox. The documentation for this is either hopelessly out of date or this is a case of abandonware. Basically all the page says is download this bridge.war and good luck. Problem #1 this bridge.war is from 1997 .. eh 2007 :-). Problem #2, I’d like to use a bit newer version of Equinox. Does this work at all? Are people still working on this? Problem #3, this page hasn’t changed substantially since I started using OSGI. Is anyone still working on this or is this a dead project? Are there any users?

Option #2 is to use Apache Felix which apparently can embed Jetty. That’s great but I’m a tomcat guy and am more interested in using tomcat as the outer container than Jetty. Neither the jetty nor the tomcat option is documented properly. I’m not even sure the tomcat option is possible/advisable. Some people hint at this being possible. A particular concern for me is that I need to cluster the damn thing, potentially on a large scale. Is this possible at all? I’m pretty sure people have done this but in terms of production quality code and documentation they have not left much of a trail. The Felix people don’t seem to much documentation in general. There’s of course the gratuitous OSGI tutorial and some hints of how you could use it but that’s it.

This situation is not something I can sell here at Nokia. I need something more substantial, preferably Tomcat or JBoss based that is 1) scalable in a cluster 2) production quality 3) well documented. I’m now pretty far convinced that what I’m looking for doesn’t exist. If I don’t find something soon, I’m going to just have to rip out all the OSGI stuff and switch to a proper dependency injecting container. Spring 3.0 is looking pretty neat for example but a bit heavyweight in my opinion.

Anyway, comments are open and please point out how wrong I am and what information I overlooked :-). My main gripe here is that I just have very little to base a decision on. Sketchy documentation, bits and pieces on blogs and mailinglists but nothing solid. Either OSGI is a genuine server side option or it is just an urban legend (some people have heard of other people that have done this). Everything I’ve seen so far hints at the latter.

I know Jboss 4, Glassfish 3, and Spring Application server are all going to be OSGI based of course. These are far from vaporware but also not exactly production ready. Additionally, being OSGI based is one thing, being able to deploy servlets from OSGI bundles is another thing. Most things I’ve read on this suggests that these servers are not really designed to allow application developers to interact with the OSGI container directly (i.e. deploying bundles, using http service instead of WAR files, etc.).

NRC Reorganization

My employer, Nokia, announced this week that it is reorganizing Nokia Research Center. The why and how of this operation is explained in the press release.

I learned this on Tuesday along with all my colleagues and have since been finding out how this will affect me. I can of course not comment on any organizational details but it is very likely that I start 2009 in a new job somewhere within Nokia since it looks like the research topic that I have been working on for the past two years is out of scope of the new Helsinki Lab research mission. While I’m of course unhappy about how that decision affects me, I accept and respect it. Short term, I am confident that I will be allowed to finish ongoing research activity since it has been so far highly successful within Nokia and we are quite close to going public with the trial of the system I demoed on Youtube a few weeks ago. I’m very motivated to do this because I’ve put a lot of time in it and want to see it succeed and get a lot of nice press attention.

However, a topic that is of course on my mind is what I will be doing after that and where I will be doing it. In short, I’m currently looking at several very interesting open positions in Nokia. Since I’m doing that anyway, I’ve decided to broaden my search and look at all available options, including those outside Nokia. I will pick the best offer I get. Don’t get me wrong, I think Nokia is a great employer and I am aware my skills are in strong demand inside Nokia. So, if Nokia makes me a good offer, I will likely accept it. But of course, the world is bigger than Finland where I have now spent three years and I am in no way geographically constrained (i.e. willing to move internationally).

So, I’ve updated my CV and am available to discuss any suitable offer.

Since this has happened in the past: please don’t contact me about Symbian programming or J2ME programming type jobs. Not interested in either, I’m a server guy.

Local Interaction demo on Youtube

I gave a demo of our system at a press event for Nokia a few weeks ago. Our PR people were busy filming and have put several short demo movies on Youtube. Including my demo and several other cool demos from colleagues in Nokia Research.

So enjoy.

Since I expect there will be people interested in learning more about this. I’ll try to give some explanation right here.

In the Youtube movie above, I am showing off our Local Interaction demo, which is a mobile website that shows off our indoor location based service platform. For the positioning we have collaborated with a different team that has been working on a indoor positioning system, which was demoed separately at the same event. Our demo leverages their technology and provides services on top.

Indoor location based services are similar to outdoor location based services in the sense that things like search, navigation, social networking, and media sharing are all things that can benefit from knowing where you are. In a nutshell, we have indoor location enhanced variants of these features integrated into our platform. However indoor location based services are different in the sense that they are much more relevant to people. People spend most of their lives indoors!

Having a platform is nice of course, but working code with real users is nicer. So we have spent most of this year preparing a trial in a real shopping mall here in Helsinki. The website you see has a nice polished UI, rich indoor content for the shopping mall and a set of useful services around the mall concept aimed at both consumers as well as proprietors of the mall, such as shop owners. The advantage of working with a real place is that it forces you to be very pragmatic about a lot of issues.

Doing a demo like ours is easy if you can assume everybody goes to the same building, uses the same phone, and just the right firmware version + your handcrafted application. A substantial part of the Ubiquitous & Pervasive Computing research community is perfectly happy with that sort of demo setups and proofs of concepts. This is why despite decades worth of demos, there’s no significant technology available for consumers beyond the boring old home automation kits and that sort of thing. It takes a bit more to make a real life impact.

A key motivation for our demo is that we don’t want to make such assumptions. Our requirements are: a capable mobile web browser (most modern Nokia phones and many phones from other vendors), and optionally, the indoor positioning client software installed. Like Location based services, our services are actually perfectly usable without positioning so we don’t actually require positioning. Being web based means that we can reach much more users than with a native application.

With everything we do we have the vision that eventually we don’t want to do this in just one shopping mall but world wide in a lot of public places. The primary goal of our trial is to gain experience rolling out this kind of technology and learning more about all the practical and technical obstacles there are for making this work on a more interesting scale in the future. We want to show that our platform is scalable in both a technical sense as well as a business sense. We want to kickoff a whole new market for indoor location based services. It doesn’t exist today, so we have to build the whole ecosystem from scratch.

For the more technical people. Our web platform is based on Python Django and integrates the positioning and other services using REST based services over HTTP. The friends feature we demo is realized using the Facebook API. We rely on Atom pub and Atom feeds internally. We intend to be mashup friendly so as to not have to reinvent every feature ourselves and instead integrate with many existing services. We have an Apache Lucene based search server that powers our indoor location based search feature. We use this feature quite heavily to look for indoor location tagged content such as photos, ads, vouchers, comments, etc. Finally, we use an off the shelf open source map server that serves up the indoor maps. In general, our philosophy is that there are already enough poorly reinvented wheels. We build what we need only if we can’t reuse what is out there. The web is out there and we use it.

For the researchers. A few articles that should be published in the next few months will outline the research we have on this. Meanwhile, you can refer to my publications page for a few workshop papers on a a predecessor of this demo that we did in 2007. Also we did a demo at the Internet of Things conference last April of our 2007 demo. And of course, there will be more details on the trial once we launch it.

About the project. This demo was developed as part of a Nokia project on an “Application Environment for Smart Spaces” which is currently running in Smart Space Lab, which is a part of Nokia Research Center. Headcount has varied quite a bit but currently we are around 8 people working full time on this. My role in this project is pushing architecture solutions and coordinating the development together with a small group of researchers and several excellent software engineers.

The Way We Live Next

I stumbled upon somebody writing about Nokia’s 2007 Way We Live Next event in Oulu. This event was intended to give the outside world a view on what is going on in Nokia Research Center.

Nice quote

Lots of interesting stuff was shown off during the course of the two days and the most interesting I came across was the indoor positioning concept. Using WiFi and specially created maps, the devices we were issued with were running the software which enabled you to move through the NRC building and pinpoint exactly where you were. So, if the next presentation was in room 101, the device would simply, and quickly show you the way. It instantly made me think of the frustration of trying to get where I want in huge shopping centres – and I figured this had to be the perfect solution.

Next week, the 2008 edition of the WWLN is going to be in Espoo and I will be giving a demo there of our indoor location based service platform, customized for a real shopping mall. We’ve demoed last years version of our software platform at the Internet of Things Conference last April. At the time our new platform was already under development for a several months and we are getting ready to start trialing it now. The WWLN event next week will be when we first show this in public and hopefully we’ll get some nice attention from the press on this.

PS. I like (good) beer …

X-plane on an iPhone

If you’ve read my past reviews of X-plane, you’ll know that I’m somewhat of a fan of that product. It’s an ultra realistic flight simulator. If you’ll read the product page you will see that it boasts a long list of features. I can assure you, they’re being modest. I don’t pay for software very often but I’ve bought v8 and recently v9 of this one.

Anyway, I just had a little WTF moment reading this little blog post from Benjamin Supnik, one of the lead developers of X-plane who works on the scenery engine. They’ve actually ported X-plane to the iPhone! If so far, you didn’t think much of the iPhone as a development platform, look again. This is really impressive.

I just looked up the product page in iTunes and it has some nice screen shots. Of course they don’t ship the full UI or scenery. All you get is an area around Inssbruck (default area in the demo version of the normal version). Probably it is heavily tuned to work nicely on an iphone. However, the mere fact that they have it running at all is very impressive. Unfortunately, it’s not possible to link to product pages in iTunes, so you’ll just have to look it up yourself, producer is Laminar or you can go to Games->Simulation in the iphone section.

Also hilareous is Austin Meyer (founder and owner of Laminar that builds X-Plane) denying there was an iphone version of x-plane, just before he launched it on September 11th. A bit of a symbolic date for launching a flight simulator, don’t you think?

Sadly, I have no iPhone or ipod touch to play with. Working for Nokia and all that.

GX WebManager

Before joining Nokia, I worked for a small web startup in the Netherlands called <GX> Creative Online Development during  2004 and 2005. When I started there, I was employee number forty something (I like to think it was 42, but not sure anymore). When I left, they had grown to close to a hundred employees and judging from what I heard since, they’ve continued to grow roughly following Moore’s law in terms of number of employees. Also they seem to have executed the strategy that took shape while I was still their release manager.

When I joined GX, GX WebManager was a pretty advanced in house developed CMS that had gone through several years of field use and evolution already and enjoyed a rapidly growing number of deployments, including many big name Dutch institutions such as KPN, Ajax, ABN-AMRO, etc. At that time it was very much a in house developed thing that nobody from outside the company ever touched. Except through the provided UI of course which was fully AJAX based before the term became fashionable. By the time I left, we had upgraded release processes to push out regular technology releases first internally and later also outside to a growing number of partners that implemented GX WebManager for their customers.

I regularly check the GX website to see what thay have been up to and recently noticed that they pushed out a community edition of GX WebManager. They’ve spent the last few years rearchitecting what was already a pretty cool CMS to begin with to refit it with a standardized content repository (JSR 170) based on Apache Jackrabbit and a OSGI container based on Apache Felix. This architecture has been designed to allow easy creation of extensions by third parties. Martijn van Berkum and Arthur Meyer (product manager and lead architect) were already musing how to do this while I was still there and had gotten pretty far doing initial designs and prototyping . Last year they pushed out GX WebManager 9.0 based on the new architecture to their partners and now 9.4 to the internet community. They seem to have pretty big ambitions to grow internationally, and in my experience the technology and know-how to do it.

So congratulations to them on completing this. If you are in the market for a CMS, go check out their products and portfolio.

Modular windows

There is a nice article on Ars discussing Microsoft’s business practices regarding windows and how they appear to be not quite working lately. It used to be that your PC came with windows whereas nowadays you have to select from a around five different versions and Microsoft is rumored to go to an even more modular and subscription based model. The general idea is to be able to squeeze out as much revenue out of the market as possible. On paper it sounds good (for MS that is).

Rather than buying an overpriced OS with everything and the kitchen sink you buy what you need. There’s a huge differences between what businesses and some individuals are willing to spend and the typical home user that just wants a browser + skype + the sims. Typically the latter group ends up buying the cheapo version and the former group ends up buying the everything and the kitchen sink version. The problem is that there is unmonetized value in the latter in the sense that some owners of the  cheapo versions might be interested in getting access to some of those features in the expensive version but not in all of them.

Now to the obvious problem with the discussed solution. By selling cheapo versions with most of the value removed and factored out into separate chunks you have to pay for, you dilute the overall value of the OS. So instead of buying an OS that can do X, Y, and Z out of the box you are buying an OS that can’t do X, Y, and Z out of the box. Marketing an OS that can’t do stuff is a lot harder than trying to sell stuff that can do things.  Worse they are opening the market to third parties that might do something similar to X, Y, and Z for a better price, or in some cases for free (beer & speech). Or even worse to companies selling an alternative OS with X, Y, and Z.

That in a nutshell is what is discussed in the Ars article and why Apple Mac OS X marketshare is approaching double digit percentages. I’ve been giving it some serious thought lately and I’m also noticing the spike in Safari users in my web site statistics.

Anyway, the reason for this write up is that the article overlooks an important argument here that I believe is relevant for more markets than just operating systems. In general, the tie between OS and features such as photo galleries, online backups, or TV UIs is artificial. Microsoft only adds features like this to make the overall OS more valuable. That is, they are looking to improve the value of the OS, not the photo gallery. However, ongoing and inevitable commoditization of software actually shifts value to new features. Especially when bundled with online subscriptions, things like online photo galleries can be quite good business. For example, Flickr has many paying subscribers.

Naturally MS is interested in markets like this (which is why they are interested in Yahoo). However, the tie-in to the OS constrains the market. Why would you not want to sell these services to Apple users? Why would you not want to sell this service to Sony Playstation owners? Why would you want to want to artificially limit who can access your service just to boost sales of your OS? As long as you were trying to artificially (and apparently for MS illegally) boost value of your core OS, bundling was a valid strategy. However, as soon as your value shifts, that becomes a brake on market growth. The OS market has commoditized to the point where you can get things like Ubuntu for free, which for the low end market is about as good as what you get with the cheapo version of Vista (see my various reviews of Ubuntu for why I’m not ready to claim better yet).

So the difference between MS and Google who is eating their lunch in the services arena is that the latter is not handicapped by 20 years of Windows legacy and can freely innovate and grow marketshare without having to worry about maintaining a revenue stream from legacy software. Google doesn’t have to sell OS licenses and so they give away software on all platforms to draw more users to their services which is where they make their money.

Naturally, Google has a lot of software engineers that are working round the clock to create more value for them. Where possible Google actively collaborates with the open source community because they know that while they won’t make any money from commodities like browsers, file systems and other important software components, they do depend on those things working as good as possible and keep evolving in the right direction. Few people appreciate this but this and not ads is why Google sponsors Firefox. It’s a brilliant strategy and it is forcing their main competitor to keep investing in internet explorer rather than being able to shift resources to directly competing with Google. 50 million $ is pocket money if it is making your main competitor crap their pants and waste resources on keeping up with you in a market where you are not even trying to make money.

You might have noticed that I have carefully avoided discussing Google and Microsoft’s mobile service strategies and also noticed that yours truly is working for Nokia. Well, my readers ought to be smart enough to figure out what I’m trying to say here aren’t you :-)?

From SPLs to Open, Compositional Platforms

Below is a position paper I submitted to the upcoming Dagstuhl seminar I am attending. It’s not peer reviewed and it is not clear at this point if there will be any proceedings. So, as an experiment, I will just put the full text in a blog post as well as the pdf you can find on my publications page. The reason I am doing this is twofold: I want people to read stuff I write and locking it up in some hard to find proceedings just isn’t doing the trick. Secondly, this blog has a comment feature. Please feel free to use it.


From SPLs to Open, Compositional Platforms

Jilles van Gurp & Christian Prehofer
Smart Space Lab
Nokia Research Center
Helsinki, Finland

Abstract. In this position paper we reflect on how software development in large organizations such as ours is slowly changing from being top down managed, as is common in SPL organizations, towards something that increasingly resembles what is happening in large open source organizations. Additionally, we highlight what this means in terms of organization and tooling.

Trends and Issues

Over the past decade of our involvement with Software Product Lines, we have seen the research field grow and prosper. By now, many companies have adopted SPL approaches for their core software development. For example, our own company, Nokia, features prominently on the SEIs Product Line hall of fame [SEI 2006]. Recently, we [Prehofer et al. 2007], and others [Ommering 2004] have published articles on the notion of compositional development that decentralizes the development of software platforms and products. The motivation for our work in this area is that we have observed that the following trends are affecting software development:

  • Widening platform scope and more diverse products. As “victims” of their own success, successful product lines allow for the creation of an ever wider range of products. Necessarily, these products have increasingly less in common with each other. Particularly, they are likely to have substantial product specific requirements and require increasing amounts of variability in the platform provided features to deal with conflicting and overlapping requirements in the base platform. In other words, the percentage of functionality shared across all products relative to the total amount of functionality in the platform is decreasing. At the same time, the percentage of platform assets actually used in any particular product is also decreasing.
  • Platforms stretch over multiple organizations. As platform and product development starts to span multiple organizational entities (companies, business units, open source projects, etc), more openness towards different and conflicting requirements, features, roadmaps and processes in different development entities is required. This concerns both open source software and commercial platforms that are developed and productized differently by third party companies.
  • Time to market and innovation speed. While time to market has always been a critical issue, it is particularly an issue with the growing size and complexity of Software Product Lines. In general, large scale software projects tend to have longer development cycles. In the case of Software Product Lines that have to cater for more and more heterogeneous products, length of development cycles
    tends to increase as complexity of the work related to defining, realizing and testing new functionality grows increasingly complex. However, time to market of features does not only include the product line development cycle but also the time needed to do product derivation as well as the development cycles of any external software the Software Product Line integrates. Worst case is that a feature first needs to be integrated in one of these dependencies; then it needs to be integrated into the next major release of the Software Product Line before finally a software product with the new feature can be developed and put in the market.

We are seeing examples of this in Nokia as well. For example, Nokia has software development spread over several major phone platforms (S30, S40, S60 and Linux Maemo) and launches multiple products from each of those platforms every year. Interesting to note here is that Nokia has never really retired a mobile phone software platform and is actively using all of them. Roughly speaking, S40 evolution is in sync with the popularization of the notion of Software Product Lines since the mid nineties. It is indeed this product line that is featured on the before mentioned SEI SPL hall of fame [SEI 2006]. Development for products and platforms is spread over many Nokia locations all over the globe as well as a complex network of subcontractors, customers and supplying companies. Additionally, the use of open source software and the intensive collaboration Nokia has with many of the associated projects are adding more complexity here. Finally, time to market is of course very important in the mobile phone market. Products tend to be on the market for only short time (e.g. 6-12 months) and developing them from a stable software platform can take more than a year in some cases. This excludes time needed for major new releases of our software platform. Consequently, disruptive new features in the platform may take years to reach the market in the form of new phones.

The way large organizations such as Nokia manage and organize their software and platform development is constantly pushing the limits of what is possible with software engineering & architecting tools and methodology. Nokia is one of a handful of companies world wide that manage tens of millions of code across its product lines and products. We see Software Product Lines as a way to develop software that has arguably been very successful in organizations like ours. However, we also note that increasingly development practice is deviating from practices that are prescribed by literature on Software Product Lines particularly with respect to centralized definition, control, ownership and management of software assets and products. Therefore, we argue that now the research community needs to adapt to this new reality as well.

The complexity and scale of the development organization increasingly make attempts to centrally manage it futile and counter productive. Conflicts of interest between stakeholders, bureaucracy, politics, etc are all affecting centralized platform and product decision making and can end up leading to unworkable compromises or delays in the software development process. Additionally, it is simply becoming impossible to develop software without depending on at least some key open source projects. Increasingly the industry is also participating as an active contributor in the open source community. Arguably, most of the open source community now consists of software developers sponsored in some way by for profit organizations. For example, Nokia is a very active participant in the mobile Linux community (the Maemo Linux platform) and ships products such as the N810 internet tablet where the majority of lines of code is actually coming from externally owned and run open source projects and even direct competitors (e.g. Intel and Motorola).

This changes the game of balancing product and platform requirements, needs and interests substantially from what is generally assumed in a classical SPL context where a single company develops both platform and products in house and where it is possibly to drive both product and platform development in a top down fashion. This simply does not work in a context where substantial amounts of critical software in a product are coming from external sources that are unwilling / unlikely to take orders from internal product managers or other types of executives external to their organization.

Effectively, this new reality necessitates a different approach to software development. Rather than driving a top down decomposition of products and features and managing development and software assets per this hierarchy, as is very much the consequence of implementing practices advertised in SPL literature, we propose to adopt a more compositional style of development.

Compositional Development

In our earlier work [Prehofer et al. 2007], we outlined an approach to adopt a more compositional approach to development. Rob van Ommering has argued along similar lines but still takes the traditional perspective of a (large) company managing a population of products [Ommering 2002][Ommering 2004]. However, what we propose here is to further decentralize development and organize similar to the open source community where many independent development teams of components, framework and product owners are working together. Each of those teams is acting to represent their own interests (and presumably those of whomever they work for). Their perspective on the external world is simply that of upstream and downstream dependencies. Downstream are the major users and customers that use the software the team produces. These stakeholders act as primary source of requirements and probably also funding. Upstream, teams operate that produce software required for using and developing the software. These teams in turn depend on their downstream dependencies and funding.

This decentralized perspective is very different from the centralized perspective and essentially allows each team to optimize for what is required from them downstream and what is available to them upstream. For example, requirements for each team come primarily from their downstream dependencies. Since there is no central controlling entity that dictates requirements, picking up these requirements and prioritizing them is very much the task of the teams themselves. Of course, they need to do so in cooperation with their downstream dependencies. Generally, especially when crossing organizational boundaries, requirements are not dictated but rather the development teams try to asses the needs of their most important customers.

Organization

As Conway’s Law [Conway 1968] predicts, the architectural decomposition of software is reflected in organizations. In many open source communities, project team dependencies reflect the architecture decomposition of software into packages, frameworks, libraries, components, or other convenient units of software decomposition. Obviously, without at least some structure and management in place, the approach advocated here results in total anarchy, which is not a good organizational model to accomplish anything but chaos. Again, we look at the open source world where organizations such as Ubuntu, Eclipse, Apache and Mozilla are driving development of thousands of projects. Each of these organizations has a surprisingly sophisticated organizational structure that comes with rules, best practices, decision making processes, etc. While there are no binding contracts enforcing these, participants in the community are required to play by the rules or risk being ignored.

In practice this means, participants voluntarily comply with practices and rules and take part in what is often called a meritocracy where important decisions are taken by those who have the merits to do so. Generally, this requires a track-record of making important contributions and having the trust of the community. For example, in the Eclipse foundation, which was founded by IBM, this means that individuals from some of their major competitors such as BEA and Red Hat actually lead some of the key projects under the eclipse umbrella. These individuals are essentially trusted by IBM to do the right things even though they work for a major competitor. Organizations such as Eclipse exist to represent the common interests of the project teams they are composed of. For example the eclipse foundation, which is very much a corporate driven (and financed) institution, represents a broad consortium of stakeholders that covers pretty much the entire spectrum of Java (and increasingly also non-Java) enterprise, desktop and mobile/embedded software related development tooling. In the past two years, they have organized two major, simultaneous releases of the major projects. In their latest release, which goes by the name of Europa, they managed to synchronize the release process of around 20 of their top level projects which are collectively developed by thousands of developers coming from dozens of companies. Many of these companies are competitors. For example, BEA and IBM are directly competing in the enterprise market and major contributors to multiple eclipse projects.

What this proves is that the way the Eclipse Foundation organizes development is extremely effective and scalable because it involves dozens of organizations and hundreds/thousands of individuals producing, integrating and testing an enormous amount of new software in a very short time frame. Organizing like this brings in the necessary flexibility to seamlessly work with numerous internal and external teams and acknowledges the reality that even internally relations between teams can be complex and difficult to manage centrally.

Tooling

A consequence of decentralizing is that aligning the use of tools across development teams becomes essential. When collaborating, it helps if tools between teams are at least similar and preferably compatible/the same. SPL research has over the past few years focused on tooling for variability management, configuration management and requirements management. However, getting these tools adopted and using them effectively in a context of thousands of software development teams that are collaborating is quite a challenge; especially since many of these tools are either in house developed or only used in a handful of companies. Tooling in the open source community tends to focus on the essentials. That being said, the OSS community has also produced many development tools that are now used on a massive scale. For example, Mozilla has had a pioneering role through their contribution of important tools such as Bugzilla and Bonsai (bug tracking and build monitoring). The whole point of the Eclipse foundation seems to be development tools. Additionally, they have a project called equinox that implements a very advanced framework that provides many interesting variability technologies and has put into mainstream use notions of using API versioning and provided and required interfaces on components. In short, there seems to be a gradual migration of SPL like tool features to mainstream tooling. Additionally, eclipse is of course a popular platform for developing such tooling in the research community.

Conclusions and Future work

In this position paper we tried to highlight a few of the key issues around the ongoing trend from integrational development towards a more open ecosystem where many stakeholders work on many pieces of software that are integrated into products by some of the stakeholders. We are currently working on an article about what it means to go from a software development practice to a compositional approach in terms of organizational models, practices and other aspects. In that article, we will list a number of practices that we associate with compositional development and evaluate these against practices in open source communities as well as selected SPL case studies from the research community. Arguably, SPLs have vastly improved software development in many companies over the past decade or so. Therefore, the key issue for the next decade will be re-aligning with the identified trends towards larger software development ecosystem while preserving and expanding the benefits that SPL development have brought.

We do not see compositional development vs. SPL development as a black and white kind of thing but instead regard this as a wide spectrum of development practices that each may or may not be applied by individual companies. The more they apply them, the more compositional their development becomes. In any case, the right set of practices is of course highly dependent on context, domain, stakeholders, etc. However, we observe that in order to scale development and in order to work with hundreds or even thousands of globally and organizationally distributed software developers effectively, it is necessary to let go of centralized control. Compositional development in this open environment is vastly more complex, organic, and so we believe, more cost effective.

References

[Conway 1968] M. E. Conway, How do committees invent, Datamation, 14(4), pp. 28-31, 1968.
[Ommering 2002] R. van Ommering, Building product populations with software components, proceedings of Proceedings of the 24rd International Conference on Software Engineering (ICSE 2002), pp. 255-265, 2002.
[Ommering 2004] R. Van Ommering, Building Product Populations with Software Components, Ph. D thesis, University of Groningen, 2004.
[Prehofer et al. 2007] C. Prehofer, J. van Gurp, J. Bosch, Compositionality in Software Platforms, in A. De Lucia, F. Ferrucci, G. Tortora, M. Tucci eds., Emerging Methods, Technologies and Process Management in Software Engineering, Wiley, 2008.
[SEI 2006] Software Engineering Institute, Product Line Hall of Fame, http://www.sei.cmu.edu/productlines/plp_hof.html, 2006.