Scrum: Agile madness

2011-12-03

Like most in our industry I’ve been exposed to agile, scrum and related hypes over the past decade. Unlike most in our industry, I have a little bit of background in software engineering research as well. During my Ph. D., which was in the early days of agile when Kent Beck published his book on extreme programming and I was writing my thesis on software architecture, I had the privilege of seeing a lot of different companies from the inside and talked to tech leads, architects and other leaders across a nice range of companies doing software product development. Ultimately after wrapping up my thesis, I decided that I lacked the experience to speak authoritatively on the subject of software development and moved over to the software industry to get some experience.

In recent years I’ve observed people making outrageous claims about the supposed benefits of things like scrum or kanban while in fact they were actively making things less agile. I’ve experienced first hand some botched roll outs of scrum. And the software engineering researcher inside of me has been itching for a while to write up my perspective on the whole thing. So, here it goes.

The word agile itself is a bit of a misnomer in the context of software development. It intends to communicate a intent to deliver software fast and efficiently. This requires that people do things in a systematic and appropriate way. That’s where the madness begins because most people will equate this to a dogmatic application of whatever software development methodology they favor. Dogmatic is in my book a synonym for stupid: you switch off your brain and blindly do whatever the dogma dictates. This is what is going on with agile software development. People go to conferences, hear opinions and sales pitches and accept them as fact. Then they go back to work and do a lot of damage.

If you read up on agile methodologies, you will find that their lead proponents (e.g. Kent Beck, Martin Fowler, Alistair Cockburn, etc.) are generally a bit more pragmatic and rather anti dogmatic than their followers and fan boys. The same goes for the proponents of scrum, including Ken Schwaber. I have great respect for these people and the fine work they have done moving the field forward.

These people are successful veteran software developers or project managers with a lot of successful projects behind them. That in my view is the main problem here: you can’t substitute experience with process and that is exactly what agile followers end up doing. Take away the experienced people and you have a lot of people executing the rituals of scrum, kanban, or whatever without really understanding what they are doing. The assumption that that will work at all is naive at best and actually rather delusional. In this article, I’ll focus mostly on scrum because that is the dominant agile methodology currently and I’ve had some first hand experience with it in my current job.

A good software organization will have key people that keep things together technically and organizationally and a larger group of people with various skills and qualities that do stuff that they’ve been told to do but are not necessarily capable of doing this unsupervised. With a good mix of those people working together, you can achieve great things fast. A bad scrum implementation will empower the latter group while taking power away from the former group. I say bad scrum implementation because in theory there are good implementations as well. It’s just I haven’t seen too many of those (i.e. none) and way too many bad ones. I believe this has a lot to do with the fact that often scrum is introduced to fix what is already a dysfunctional organization.

In the scrum world you have a product owner representing the business side who negotiates with the software development team about features and a self organized team delivering ‘business value’ as fast as they can with a scrum master that acts as a facilitator. In this happy utopia everybody is happy and productive. This works well under the following assumptions:

  1. The product owner has a firm grasp on domain and the technology.
  2. The team can self organize and there is a balance between team empowerment and product owner power.
  3. Everybody involved understands what they are supposed to do in their scrum role
  4. The amount of work is small enough to be handled by one team

Actually, if all of the above is true, you will have little need for process. If that isn’t the case, no amount of process is going to fix things magically. In my experience, all of these assumptions are generally broken in scrum organizations. The reality is that product owners are rather well aligned with management and whatever they say ends up being an executive order. It’s elementary corporate picking order. Additionally, big organizations have many teams and consequently many product owners. Scrum in combination with a full blown corporate rat race like that is going to have some very obvious consequences. Including management not agreeing amongst themselves what needs to be done and fighting over the heads of individual teams about features, products, vision and direction. Ultimately this results in mediocre products and failure in the market.

Teams self organize to do whatever the hell they are told to do by the product owners, who with the best intentions are probably demanding the wrong things to be built on unrealistically short notice due to a fundamental lack of a grasp on either the domain, the technology, and the basics of software project management. The scrum masters basically get the shitty job of keeping it together and getting the teams to ‘democratically’ agree on stuff while not actually having any decision power to make them do the right thing. Generally the scrum master should be one of the more senior team members that would in a traditional organization have been either a project manager (with full power) or a tech lead/architect with decision powers.

I’ve seen some spectacularly mediocre software delivered this way. It’s not pretty. The problem is usually a lack of experience that gets amplified by the process. A two day pep talk about scrum is apparently all the training you need to become a product manager or scrum master. I know because I’m a certified scrum master (not practicing it though) and I know how hollow that phrase is.I congratulate Ken Schwaber on his well oiled business of scrum training large parts of the industry (me included) but I believe that he doesn’t honestly believe a two day training is enough either.

The role of product owner is generally a first step up the corporate ladder. In other words, there are a lot of inexperienced people who get promoted to be a product owner or worse start their first or second job being one. Likewise, the scrum master role is not considered a job very high on the corporate ladder either. Think developers with a few years of experience who suddenly find themselves moving post its around on walls rather than doing the stuff they were trained to do (i.e. software development). How can you expect inexperienced people like this to succeed?

Well you can’t and that is why scrum isn’t working. You can’t fix a dysfunctional software organization by rolling out scrum and mass promoting people to be product managers or scrum masters. It doesn’t work this way but yet this is exactly what happens when big companies change process. It works as follows: you spend a gazillion of dollars on training and consultancy to force the organization in a particular direction. Naturally, you want results fast so you cut some corners. And before you know it the walls of your offices are plastered with colorful post-its and the meeting rooms are full of busy looking people doing retrospectives, sprint plannings, estimation sessions, etc. Reports on velocity and story points get exchanged and proud product owners brag about their products. It’s no wonder executives love scrum: they get the illusion of progress.

The problem is that all you have at this point is a lot of people going through the moves of a process without actually delivering any coherent results whatsoever. Getting one team to deliver something is relatively easy. Getting 50 teams to deliver something coherent together is a different matter. The natural reflex in big organizations is to organize hierarchically and in a scrum context this means that product managers report to super product managers who report to department managers who report to VPs who report to the CEO who wants results yesterday. So when problems happen, they are escalated up and translated into executive orders that travel down. When it comes to solving technical problems, having a lot of non technical people involved doesn’t necessarily make things easier. So, the organization becomes a bottleneck for doing things the right way. Technical issues are solved relatively easy if you stick the engineers involved in one room. But that doesn’t happen if you add layers of management with conflicting interests that need to approve, be involved, escalate, etc. Even getting them to understand the problem is hard.

Another problem is that teams have dependencies. Team A needs the stuff of team B who needs the stuff of team C, etc. Any good software architect knows how to manage dependencies between components. But your average scrum product owner is not an architect. So, you get organizational complexity (think circular dependencies between teams) and organization driven architecture where teams raise fences (APIs, separate deployment processes, etc.) to shield them from neighboring teams. To make things worse, scrum is all about leaving teams alone during their sprints so you have a lot of teams delivering stuff every sprint and throwing it over the fence to other teams at the end. Orchestrating the development of a few dozen scrum teams is a hard problem and I’m pretty sure very few organizations have actually solved this problem. Meanwhile all the scrum training seems to be geared towards teams operating in isolation.

Is there a happy ending here? Sure, eventually inexperienced people become experienced people. Experienced people tend to do things that they know have worked for them in the past and challenge things that they know to be flawed. This includes taking control, leading and doing things in a way they believe will deliver results rather than in a way that somebody else claims will deliver results. The fallacy of scrum and agile is that initially you have a lot of people in the latter group and very few in the former.

Over the past seven years, I’ve seen scrum penetrate the minds of senior executives who then pushed for corporate wide rollout of this great new process (a well known anti pattern for scrum that Ken Schwaber warns for in his courses) that would magically fix things. Lots of product owners and scrum masters were hired/trained/promoted and then the obvious things started happening: large amounts of teams delivering the wrong software too late and not aligning and synchronizing with each other at all.

But then other things started to happen. We got some veteran, experienced software developers in some key positions in the company. They started fixing things; they started removing bureaucratic obstacles, dealing with the technical debt, and they started emphasizing good practices such as accountability, technical skill, and favoring pragmatism over dogmatism. I would say by this point, scrum is definitely on the way out. We still have floors where the walls are still plastered in post its. But we also have other floors where the walls are now empty but are still bearing the traces of what used to be scrum boards and where engineers work on delivering great software without too much interference from random management. That’s fine. Choosing and tuning the way in which you work is more important than having everybody mindlessly follow the same process.

The story of Nokia (where I work), which is often cited as an early adopter of scrum, has been well publicized and clearly, we’ve had a huge software problem over the past years. After a few bad years, Nokia is now delivering good, solid products again. The recent product announcements like the Lumia phones and Nokia Drive are good examples of a software development organization delivering results fast (i.e. truly agile). Few people appreciate how little time has actually passed between the initial announcements early this year and us delivering the goods this summer. Getting any organization this large too deliver that fast is kind of impressive. I’m afraid I wouldn’t give scrum much credit for it.

In the end what matters is having a good mix of people with experience, skills and vision. Inexperienced people learn quick in such an environment and good people get rewarded for good work.