Scrum: Agile madness

Like most in our industry I’ve been exposed to agile, scrum and related hypes over the past decade. Unlike most in our industry, I have a little bit of background in software engineering research as well. During my Ph. D., which was in the early days of agile when Kent Beck published his book on extreme programming and I was writing my thesis on software architecture, I had the privilege of seeing a lot of different companies from the inside and talked to tech leads, architects and other leaders across a nice range of companies doing software product development. Ultimately after wrapping up my thesis, I decided that I lacked the experience to speak authoritatively on the subject of software development and moved over to the software industry to get some experience.

In recent years I’ve observed people making outrageous claims about the supposed benefits of things like scrum or kanban while in fact they were actively making things less agile. I’ve experienced first hand some botched roll outs of scrum. And the software engineering researcher inside of me has been itching for a while to write up my perspective on the whole thing. So, here it goes.

The word agile itself is a bit of a misnomer in the context of software development. It intends to communicate a intent to deliver software fast and efficiently. This requires that people do things in a systematic and appropriate way. That’s where the madness begins because most people will equate this to a dogmatic application of whatever software development methodology they favor. Dogmatic is in my book a synonym for stupid: you switch off your brain and blindly do whatever the dogma dictates. This is what is going on with agile software development. People go to conferences, hear opinions and sales pitches and accept them as fact. Then they go back to work and do a lot of damage.

If you read up on agile methodologies, you will find that their lead proponents (e.g. Kent Beck, Martin Fowler, Alistair Cockburn, etc.) are generally a bit more pragmatic and rather anti dogmatic than their followers and fan boys. The same goes for the proponents of scrum, including Ken Schwaber. I have great respect for these people and the fine work they have done moving the field forward.

These people are successful veteran software developers or project managers with a lot of successful projects behind them. That in my view is the main problem here: you can’t substitute experience with process and that is exactly what agile followers end up doing. Take away the experienced people and you have a lot of people executing the rituals of scrum, kanban, or whatever without really understanding what they are doing. The assumption that that will work at all is naive at best and actually rather delusional. In this article, I’ll focus mostly on scrum because that is the dominant agile methodology currently and I’ve had some first hand experience with it in my current job.

A good software organization will have key people that keep things together technically and organizationally and a larger group of people with various skills and qualities that do stuff that they’ve been told to do but are not necessarily capable of doing this unsupervised. With a good mix of those people working together, you can achieve great things fast. A bad scrum implementation will empower the latter group while taking power away from the former group. I say bad scrum implementation because in theory there are good implementations as well. It’s just I haven’t seen too many of those (i.e. none) and way too many bad ones. I believe this has a lot to do with the fact that often scrum is introduced to fix what is already a dysfunctional organization.

In the scrum world you have a product owner representing the business side who negotiates with the software development team about features and a self organized team delivering ‘business value’ as fast as they can with a scrum master that acts as a facilitator. In this happy utopia everybody is happy and productive. This works well under the following assumptions:

  1. The product owner has a firm grasp on domain and the technology.
  2. The team can self organize and there is a balance between team empowerment and product owner power.
  3. Everybody involved understands what they are supposed to do in their scrum role
  4. The amount of work is small enough to be handled by one team

Actually, if all of the above is true, you will have little need for process. If that isn’t the case, no amount of process is going to fix things magically. In my experience, all of these assumptions are generally broken in scrum organizations. The reality is that product owners are rather well aligned with management and whatever they say ends up being an executive order. It’s elementary corporate picking order. Additionally, big organizations have many teams and consequently many product owners. Scrum in combination with a full blown corporate rat race like that is going to have some very obvious consequences. Including management not agreeing amongst themselves what needs to be done and fighting over the heads of individual teams about features, products, vision and direction. Ultimately this results in mediocre products and failure in the market.

Teams self organize to do whatever the hell they are told to do by the product owners, who with the best intentions are probably demanding the wrong things to be built on unrealistically short notice due to a fundamental lack of a grasp on either the domain, the technology, and the basics of software project management. The scrum masters basically get the shitty job of keeping it together and getting the teams to ‘democratically’ agree on stuff while not actually having any decision power to make them do the right thing. Generally the scrum master should be one of the more senior team members that would in a traditional organization have been either a project manager (with full power) or a tech lead/architect with decision powers.

I’ve seen some spectacularly mediocre software delivered this way. It’s not pretty. The problem is usually a lack of experience that gets amplified by the process. A two day pep talk about scrum is apparently all the training you need to become a product manager or scrum master. I know because I’m a certified scrum master (not practicing it though) and I know how hollow that phrase is.I congratulate Ken Schwaber on his well oiled business of scrum training large parts of the industry (me included) but I believe that he doesn’t honestly believe a two day training is enough either.

The role of product owner is generally a first step up the corporate ladder. In other words, there are a lot of inexperienced people who get promoted to be a product owner or worse start their first or second job being one. Likewise, the scrum master role is not considered a job very high on the corporate ladder either. Think developers with a few years of experience who suddenly find themselves moving post its around on walls rather than doing the stuff they were trained to do (i.e. software development). How can you expect inexperienced people like this to succeed?

Well you can’t and that is why scrum isn’t working. You can’t fix a dysfunctional software organization by rolling out scrum and mass promoting people to be product managers or scrum masters. It doesn’t work this way but yet this is exactly what happens when big companies change process. It works as follows: you spend a gazillion of dollars on training and consultancy to force the organization in a particular direction. Naturally, you want results fast so you cut some corners. And before you know it the walls of your offices are plastered with colorful post-its and the meeting rooms are full of busy looking people doing retrospectives, sprint plannings, estimation sessions, etc. Reports on velocity and story points get exchanged and proud product owners brag about their products. It’s no wonder executives love scrum: they get the illusion of progress.

The problem is that all you have at this point is a lot of people going through the moves of a process without actually delivering any coherent results whatsoever. Getting one team to deliver something is relatively easy. Getting 50 teams to deliver something coherent together is a different matter. The natural reflex in big organizations is to organize hierarchically and in a scrum context this means that product managers report to super product managers who report to department managers who report to VPs who report to the CEO who wants results yesterday. So when problems happen, they are escalated up and translated into executive orders that travel down. When it comes to solving technical problems, having a lot of non technical people involved doesn’t necessarily make things easier. So, the organization becomes a bottleneck for doing things the right way. Technical issues are solved relatively easy if you stick the engineers involved in one room. But that doesn’t happen if you add layers of management with conflicting interests that need to approve, be involved, escalate, etc. Even getting them to understand the problem is hard.

Another problem is that teams have dependencies. Team A needs the stuff of team B who needs the stuff of team C, etc. Any good software architect knows how to manage dependencies between components. But your average scrum product owner is not an architect. So, you get organizational complexity (think circular dependencies between teams) and organization driven architecture where teams raise fences (APIs, separate deployment processes, etc.) to shield them from neighboring teams. To make things worse, scrum is all about leaving teams alone during their sprints so you have a lot of teams delivering stuff every sprint and throwing it over the fence to other teams at the end. Orchestrating the development of a few dozen scrum teams is a hard problem and I’m pretty sure very few organizations have actually solved this problem. Meanwhile all the scrum training seems to be geared towards teams operating in isolation.

Is there a happy ending here? Sure, eventually inexperienced people become experienced people. Experienced people tend to do things that they know have worked for them in the past and challenge things that they know to be flawed. This includes taking control, leading and doing things in a way they believe will deliver results rather than in a way that somebody else claims will deliver results. The fallacy of scrum and agile is that initially you have a lot of people in the latter group and very few in the former.

Over the past seven years, I’ve seen scrum penetrate the minds of senior executives who then pushed for corporate wide rollout of this great new process (a well known anti pattern for scrum that Ken Schwaber warns for in his courses) that would magically fix things. Lots of product owners and scrum masters were hired/trained/promoted and then the obvious things started happening: large amounts of teams delivering the wrong software too late and not aligning and synchronizing with each other at all.

But then other things started to happen. We got some veteran, experienced software developers in some key positions in the company. They started fixing things; they started removing bureaucratic obstacles, dealing with the technical debt, and they started emphasizing good practices such as accountability, technical skill, and favoring pragmatism over dogmatism. I would say by this point, scrum is definitely on the way out. We still have floors where the walls are still plastered in post its. But we also have other floors where the walls are now empty but are still bearing the traces of what used to be scrum boards and where engineers work on delivering great software without too much interference from random management. That’s fine. Choosing and tuning the way in which you work is more important than having everybody mindlessly follow the same process.

The story of Nokia (where I work), which is often cited as an early adopter of scrum, has been well publicized and clearly, we’ve had a huge software problem over the past years. After a few bad years, Nokia is now delivering good, solid products again. The recent product announcements like the Lumia phones and Nokia Drive are good examples of a software development organization delivering results fast (i.e. truly agile). Few people appreciate how little time has actually passed between the initial announcements early this year and us delivering the goods this summer. Getting any organization this large too deliver that fast is kind of impressive. I’m afraid I wouldn’t give scrum much credit for it.

In the end what matters is having a good mix of people with experience, skills and vision. Inexperienced people learn quick in such an environment and good people get rewarded for good work.

16 Replies to “Scrum: Agile madness”

  1. You are being hypocritical.

    On the one hand you are accusing your managers of seeking salvation in The Process without tinkering with the actual problems. On the other hand you are expecting salvation by The Process without tinkering with the actual problems yourself.

    SCRUM is a tool which provides swift feedback on how you are doing, not unlike your compiler gives feedback about syntactical mistakes in your code, or your unit tests provide information about semantic mistakes, or your continuous integration / continuous deployment points you to integration issues. 

    SCRUM is a huge help when it comes to identifying procedural problems, deficiencies, work overload, or the like. Of course you have to make use of that feedback. You’ve got to deal with the problems yourself. Your annoying little rant however does suggest a process (specifically SCRUM) should actually fix things for you.

    1. Scrum is being pitched as a silver bullet on a massive scale in our industry and the industry seems to be blindly in love with it now. My point here is that the way the industry is blindly applying scrum everywhere is a form of madness that is very harmful.

      I know the theory and I have seen the practice. They don’t line up and there is very little acknowledgement of the cold hard empirical facts here of scrum in practice. Now the standard fanboy answer to any criticism is “you’re not doing it properly”. Well sorry, but I’m calling that out as bull shit. There seems to be a whole lot “not doing it properly” going on in the industry and I have seen 0 credible empirical evidence of people doing it properly doing any good either. It seems the success is always somewhere else. I’ve seen plenty of over priced consultants making good money out of selling this snake oil but preciously little success for it.

      Having seen how credible empirical evidence is supposed to look before: the average scrum sales pitch aint it. Not even close. I’ve read the research papers and they are excellent studies of very specific situations that only a fool would extrapolate as generally applicable. You will find no such claim in those papers BTW: it would have blocked publication since you can’t have baseless claims in a serious publication. People confusing sales pitch with evidence is a big part of the problem. You certainly seem to have bought into it. You being on a successful project proves exactly nothing. It only proves you have completed a project successfully. Probably the skills of the people around you were a massive factor in that success.

      Now I’ve been in the industry long enough to see silver bullets come and go. Scrum is past its prime in that sense and I fully expect people to move on very soon. Consultants have cashed their money and they need new snake oil to sell very soon. Every silly consulting shop is doing scrum these days (or claiming to) and last time I checked the number of failing IT projects continues to be depressingly high. Scrum or no scrum.

      1. Yes, the Scrum Master has no clothes, it’s time to end the cult of agile and get on with common sense.

  2. I agree with Thilo in as far as Scrum *is* just a process, not a magical problem solver.   My understanding of Scrum’s basic promise is :  it unearths problems, but doesn’t deal with them. The people in the team are the problem solvers at the end of the day. Strong technical people, and managers (allbeit in whatever guise that takes in the world of Agile) are still a must have, as is accountability.

    Relying on Scrum by itself, in the hope it will solve problems, clearly is not going to deliver anything bar colourful boards showing nothing is “done”. Yes, implemented incorrectly (I choose that word on purpose), it can even provide a cover for mediocrity. How often have we heard the phrase “…but thats not Agile!” get uttered by people trying to deflect or else just hide from the truth of the matter?

    I’ve seen Scrum deliver on the promise mentioned at the start of my comment however – and teams happily and repeatedly deliver real business value. Maybe I’ve just been lucky, yes, but I think luck wasn’t such a big factor in these cases – people got into the right mindset and observed the spirit of the approach.  The point about it being used as a cure-all for dysfunctional dev orgs being the road to disaster is completely valid, but labelling Scrum “madness” is a bit blinkered, in my humble opinion.

    1. I’m well aware of the disclaimers that scrum comes with. The point is, most people implementing and selling it seem to conveniently overlook those. So, yes: technically you can’t blame scrum. But I do blame the hordes of overpriced consultants for doing an awful job of selling and implementing it across our industry. I don’t blame the tool: I blame the people using it as the proverbial hammer for wacking every nail in sight.

      As stated above, observing success in isolated cases proves nothing whatsoever. I’m sure that if you do some honest reflection, you can come up with a couple of failures as well. And you probably learned from those failures. That’s called experience and my central claim above is that experience trumps process.

      If you need more evidence: make a list of the most important software in your daily life (developer tooling, OS, embedded software in your microwave, etc.). Now filter that list by those doing scrum properly. QED: you can deliver good software without scrum. In fact, that’s how most software gets written.

  3. “You can’t substitute experience with process.”

    That’s the problem in a nutshell. Well said.

  4. Jilles, thank you for these thoughts. I think you have made very valid points. Like Dave Nicolette, I think “You can’t substitute experience with process” is one of the most important things we should all remember.

  5. ‘It’s no wonder executives love scrum: they get the illusion of progress.’ hits the nail right on the head.

    Silo’d scrum teams isolated from the outside world during their sprint leads to the worst corporate behaviour – friction between teams, politics up the ladder and across inter-dependent teams, petty squabbling within teams as pecking orders in stand-ups are established and forward thinking junior coders are pushed down (‘don’t gold plate’ seems to be the final word in mediocracy) and the need to ‘sell’ software progress to management takes over actually delivering what a business NEEDS (which isn’t always what they asked for). Contesting requirements is a capital offence to a lowly coder. Talking to the business without the product owner is just plain madness.

    I hate scrum and I hate agile. Its the lazy mans way to justify software, its NOT the passionate developers way to develop great solutions.

  6. Thank you very much for your concise and illuminating article. I hope some day scrum with its rigid dogmatism will come down like the Berlin Wall.

  7. Very well said. There’s one thing I like about scrum, it’s daily stand. I’ve seen developers days on end without reporting progress or problems. The daily stand (or whatever it is called these days) allows a team an easier way to monitor, help and steer new team members and juniors. It seems to lower the boundries between diciplines and experiences.

  8. I was highly surprised when I had a first stand-up. People were literally standing! This a new religion! I experienced the daily meetings as demotivating as my weekly visits on sunday to the church as a child, and most of the time a waste. Most productive are ad-hoc meetings that you can have when you want to create a solution for a problem together. As Andy hereabove says “its NOT the passionate developers way to develop great solutions”.

  9. I couldn’t agree more Jilles. The religious adherence to agile/scrum ‘ceremonies’ is scary in that new grads must worship at the altar or they won’t get a jib, never mind get on afterwards. Anyone who grew up in the 80s and worked in the SW community in the 90s has seem multiple frameworks & methodologies for process-izing common sense. Thuis is the bottom line – it strats with common sense. you must have a experienced people. That’s the starting point. The way points and poker and all this rubbish has become ‘normal’ just astounds me – it seems the newer generations ahve never had to deep dive things to get a solid grip on the functionality required. Incredible some of the agile proponents and how inflexible they are and how this must* adhere to *their* version of agile. Sigh…religion again only offering false hope…

Leave a Reply