The Secret Reason that Development Teams Struggle to Deliver On Time, On Budget, or On Scope

How long will it take you to run to the store and buy some milk?

You know how far away the store is, you know where they keep the milk, and you know that you’ll probably have to spend a few minutes checking out. But, chances are this isn’t enough information to answer the question.

If you’re like most people, you’re terrible at these kinds of predictions.

You ignore all of the small details. You overestimate your ability to get in and get out. You underestimate the time it will take to get through traffic.

So, whatever number popped into your head–say, 10 minutes?–is almost certainly wrong.

Psychologists call this The Planning Fallacy. It’s a common problem that we all face.

When asked to estimate how long something will take, we always tend to get it wrong. And this leads to a lot of issues in both our personal and professional lives.

So what’s to be done?

First, we must understand the root issues at play when it comes to The Planning Fallacy and then we can start to unravel ways to stop it from happening.

Two Types of Thinking: One is Really Bad at Planning

At a basic level, humans have two “types” of thinking, or systems that we use for different thinking tasks.

System 1 thinking is categorized as being quick, intuitive, and effortless. It’s the system that you use when you look at a string of numbers and see patterns or repetition.

If I ask you to solve 2 * 2, you would, of course, have a quick answer.

Although it probably doesn’t seem like you had to do much thinking, your brain processed the request, understood the pattern, and retrieved the stored answer—that’s System 1 at work.

System 2 thinking is effortful. It’s the kind of thinking that you do when you’re analyzing a large quantity of data or working through a complex math problem.

If I asked you to solve 123 * 48, you could probably do it. But it wouldn’t come to mind quickly.

You’d need to mentally process this problem in a way that feels much more “manual” and less intuitive. This is emblematic of System 2, which usually only engages when your System 1 can’t come up with a quick and intuitive answer.

In his book, “Thinking Fast and Slow”, Daniel Kahneman calls System 2 “lazy”. Or, maybe more accurately, we humans are generally lazy about deploying System 2 thinking because it requires more deliberate effort and concentration.

So, we often defer to System 1–intuition–even when we shouldn’t.

This is why we’re bad at planning.

If I ask you how long it will take you accomplish a task, which kind of thinking will you use?

Almost everyone relies on System 1. They consider the task for a brief second, make a mental approximation, and come up with a response that seems intuitive and reasonable.

Surely it won’t take you more than 10 minutes to run to the store and buy milk, right?

Of course, anyone who has had to make a last-minute run for milk should know—from experience—that it’s almost never as simple as what the intuitive answer would lead us to believe.

Instead, there are nuances we must consider. Like, the fact that there may be only one cashier working at this time, which means you’ll have to wait in line to check out. Or, the fact that there’s construction on Colfax Avenue, which means it will take you 10 minutes to get to the store instead of 5.

This is where System 2 comes in—if we performed a more thorough analysis, these factors would have been considered in our answer. Then it would be clear that it’s much more likely to take 20 or 30 minutes to run to the store instead of 10.

But, we rarely do this when it comes to planning. Why?

The High Costs of Poor Planning

In most day-to-day decisions, we are perfectly content to make simple, intuitive estimates like this. There is usually very little consequence to being wrong (other than being a bit late).

But when it comes to work—and software development, in particular—the cost for poor planning can be enormous.

Teams that perform poorly at planning are more likely to blow through budgets, miss deadlines, and have a generally unpleasant feeling about the work they’re doing.

Estimates show that failed software projects (including those that are late or fail to meet scope expectations) cost between $50-80 billion annually.

But, more acutely, poor planning creates unnecessary stress, anxiety, and problems within any company. When projects run off the rails, teams are left to pick up the pieces. Meanwhile, technical debt adds up, which creates a cascade effect on future projects, and blown timelines compound as the team is constantly trying to catch up.

Being behind sucks. Constantly being behind sucks even more.

Avoiding the Planning Fallacy in Software Development

The Planning Fallacy pops up everywhere. From planning a birthday party to your commute, you’ll notice that any time you’re asking to give a time estimate, you’re likely not very good at being accurate.

But, it’s an especially difficult and persistent problem when it comes to software development. Because planning in software projects is often complex and involves multiple contiguous estimates, the results of those estimates being wrong will compound over time.

The project that your team thought would take 3 weeks ends up taking 6 weeks.

And because that project took twice as long as expected, every other planned project must get pushed back. But, to make matters worse, the entire team is fighting the urge to just push the thing out the door, which leads to mistakes, oversights, and shortcuts (AKA technical debt). So, then you have even more to clean up.

In a word, it’s frustrating.

But you can change this. By understanding The Planning Fallacy and adopting strategies and processes to overcome it, your team can plan more accurately and avoid the headaches that come with failed sprints and blown budgets.

1. Devote the proper time to planning

It’s no secret that software teams tend to hate meetings. And many planning sessions are needlessly monotonous. But one of the main reasons that teams hate planning so much is because they’re doing it poorly.

It seems like a waste of time to go through the planning process when the plan is never accurate anyway.

Of course, this doesn’t have to be the case. If your team works to improve the planning process–to provide better, more reasonable estimates and timelines–then planning becomes a valuable part of the software development cycle.

And it’s important to do this.

Having a clear plan for each sprint will make everyone’s life easier. And carving out time for sprint planning is critical to setting up the team for success.

2. Plan based on data, not intuition

Probably the most important aspect of this entire process is to shift from using System 1 thinking (intuition) to System 2 thinking (data).

What this means in practice is to take the time to analyze the work that’s being presented and then put together an estimate based on something other than gut feeling.

This is one benefit of using time tracking software (or implementing systems similar to a GitHub project management tool).

If the team is scoping out the sprint to complete a certain type of project, then they can reference past projects of similar type or complexity to get an empirical understanding of how long it actually took.

This will let the team provide better estimates and/or compare their estimates versus past reality to see if they line up.

Including this step in the planning process will force the team out of System 1 thinking and into a thoughtful, System 2-driven analysis of the actual data at hand.

So, you’ll end up with better, smarter, and more accurate estimates.

3. Add a buffer (of course)

Look, sh*t happens.

Even the best-laid plans run awry. And, no matter how much thought and effort you put into your planning efforts, there is no way to plan for the unexpected. Most teams understand this and already add some kind of a buffer into their planning.

The problem is that many teams use this as their only tool for combating The Planning Fallacy. It’s the ol’, “take your estimate and double it,” approach

While it may work on occasion, this approach is scattershot and imprecise. It leads to other problems because you aren’t addressing the core reason why the team fails to plan effectively in the first place.

So it’s important to combine this buffer with smarter planning practices to get more precise and realistic understanding of time and work.

Individually, none of these practices will radically change anything. But, together in a proper process and system for planning, they become a powerful weapon.