GET STARTED
Blog Menu
Published:Jun 04, 2018

The Secret Reason that Development Teams Struggle to Deliver On Time, On Budget, or On Scope

How long will it take you to run to the store and buy some milk?

You know how far away the store is, you know where they keep the milk, and you know that you’ll probably have to spend a few minutes checking out. But, chances are this isn’t enough information to answer the question.

If you’re like most people, you’re terrible at these kinds of predictions.

You ignore all of the small details. You overestimate your ability to get in and get out. You underestimate the time it will take to get through traffic.

So, whatever number popped into your head–say, 10 minutes?–is almost certainly wrong.

Psychologists call this The Planning Fallacy. It’s a common problem that we all face.

When asked to estimate how long something will take, we always tend to get it wrong. And this leads to a lot of issues in both our personal and professional lives.

So what’s to be done?

First, we must understand the root issues at play when it comes to The Planning Fallacy and then we can start to unravel ways to stop it from happening.

Two Types of Thinking: One is Really Bad at Planning

At a basic level, humans have two “types” of thinking, or systems that we use for different thinking tasks.

System 1 thinking is categorized as being quick, intuitive, and effortless. It’s the system that you use when you look at a string of numbers and see patterns or repetition.

If I ask you to solve 2 * 2, you would, of course, have a quick answer.

Although it probably doesn’t seem like you had to do much thinking, your brain processed the request, understood the pattern, and retrieved the stored answer—that’s System 1 at work.

System 2 thinking is effortful. It’s the kind of thinking that you do when you’re analyzing a large quantity of data or working through a complex math problem.

If I asked you to solve 123 * 48, you could probably do it. But it wouldn’t come to mind quickly.

You’d need to mentally process this problem in a way that feels much more “manual” and less intuitive. This is emblematic of System 2, which usually only engages when your System 1 can’t come up with a quick and intuitive answer.

In his book, “Thinking Fast and Slow”, Daniel Kahneman calls System 2 “lazy”. Or, maybe more accurately, we humans are generally lazy about deploying System 2 thinking because it requires more deliberate effort and concentration.

So, we often defer to System 1–intuition–even when we shouldn’t.

This is why we’re bad at planning.

If I ask you how long it will take you accomplish a task, which kind of thinking will you use?

Almost everyone relies on System 1. They consider the task for a brief second, make a mental approximation, and come up with a response that seems intuitive and reasonable.

Surely it won’t take you more than 10 minutes to run to the store and buy milk, right?

Of course, anyone who has had to make a last-minute run for milk should know—from experience—that it’s almost never as simple as what the intuitive answer would lead us to believe.

Instead, there are nuances we must consider. Like, the fact that there may be only one cashier working at this time, which means you’ll have to wait in line to check out. Or, the fact that there’s construction on Colfax Avenue, which means it will take you 10 minutes to get to the store instead of 5.

This is where System 2 comes in—if we performed a more thorough analysis, these factors would have been considered in our answer. Then it would be clear that it’s much more likely to take 20 or 30 minutes to run to the store instead of 10.

But, we rarely do this when it comes to planning. Why?

The High Costs of Poor Planning

In most day-to-day decisions, we are perfectly content to make simple, intuitive estimates like this. There is usually very little consequence to being wrong (other than being a bit late).

But when it comes to work—and software development, in particular—the cost for poor planning can be enormous.

Teams that perform poorly at planning are more likely to blow through budgets, miss deadlines, and have a generally unpleasant feeling about the work they’re doing.

Estimates show that failed software projects (including those that are late or fail to meet scope expectations) cost between $50-80 billion annually.

But, more acutely, poor planning creates unnecessary stress, anxiety, and problems within any company. When projects run off the rails, teams are left to pick up the pieces. Meanwhile, technical debt adds up, which creates a cascade effect on future projects, and blown timelines compound as the team is constantly trying to catch up.

Being behind sucks. Constantly being behind sucks even more.

Avoiding the Planning Fallacy in Software Development

The Planning Fallacy pops up everywhere. From planning a birthday party to your commute, you’ll notice that any time you’re asking to give a time estimate, you’re likely not very good at being accurate.

But, it’s an especially difficult and persistent problem when it comes to software development. Because planning in software projects is often complex and involves multiple contiguous estimates, the results of those estimates being wrong will compound over time.

The project that your team thought would take 3 weeks ends up taking 6 weeks.

And because that project took twice as long as expected, every other planned project must get pushed back. But, to make matters worse, the entire team is fighting the urge to just push the thing out the door, which leads to mistakes, oversights, and shortcuts (AKA technical debt). So, then you have even more to clean up.

In a word, it’s frustrating.

But you can change this. By understanding The Planning Fallacy and adopting strategies and processes to overcome it, your team can plan more accurately and avoid the headaches that come with failed sprints and blown budgets.

1. Devote the proper time to planning

It’s no secret that software teams tend to hate meetings. And many planning sessions are needlessly monotonous. But one of the main reasons that teams hate planning so much is because they’re doing it poorly.

It seems like a waste of time to go through the planning process when the plan is never accurate anyway.

Of course, this doesn’t have to be the case. If your team works to improve the planning process–to provide better, more reasonable estimates and timelines–then planning becomes a valuable part of the software development cycle.

And it’s important to do this.

Having a clear plan for each sprint will make everyone’s life easier. And carving out time for sprint planning is critical to setting up the team for success.

2. Plan based on data, not intuition

Probably the most important aspect of this entire process is to shift from using System 1 thinking (intuition) to System 2 thinking (data).

What this means in practice is to take the time to analyze the work that’s being presented and then put together an estimate based on something other than gut feeling.

This is one benefit of using time tracking software.

If the team is scoping out the sprint to complete a certain type of project, then they can reference past projects of similar type or complexity to get an empirical understanding of how long it actually took.

This will let the team provide better estimates and/or compare their estimates versus past reality to see if they line up.

Including this step in the planning process will force the team out of System 1 thinking and into a thoughtful, System 2-driven analysis of the actual data at hand.

So, you’ll end up with better, smarter, and more accurate estimates.

3. Add a buffer (of course)

Look, sh*t happens.

Even the best-laid plans run awry. And, no matter how much thought and effort you put into your planning efforts, there is no way to plan for the unexpected. Most teams understand this and already add some kind of a buffer into their planning.

The problem is that many teams use this as their only tool for combating The Planning Fallacy. It’s the ol’, “take your estimate and double it,” approach

While it may work on occasion, this approach is scattershot and imprecise. It leads to other problems because you aren’t addressing the core reason why the team fails to plan effectively in the first place.

So it’s important to combine this buffer with smarter planning practices to get more precise and realistic understanding of time and work.

Individually, none of these practices will radically change anything. But, together in a proper process and system for planning, they become a powerful weapon.

 

Free eBook

Rethinking Timekeeping for Developers:

Turning a Timesuck Into Time Well Spent

16 comments

New top story on Hacker News: Why Development Teams Struggle to Deliver on Time, on Budget, or at All – techofacts

06-05-2018 14:00

[…] Why Development Teams Struggle to Deliver on Time, on Budget, or at All 3 by encorekt | 0 comments on Hacker News. […]

New top story on Hacker News: Why Development Teams Struggle to Deliver on Time, on Budget, or at All - EYFnews

06-05-2018 14:00

[…] Why Development Teams Struggle to Deliver on Time, on Budget, or at All 3 by encorekt | 0 comments on Hacker News. […]

Why Development Teams Struggle to Deliver on Time, on Budget, or at All - newsfest

06-05-2018 14:01

[…] Why Development Teams Struggle to Deliver on Time, on Budget, or at All 3 by encorekt | 0 comments on Hacker News. […]

New top story on Hacker News: Why Development Teams Struggle to Deliver on Time, on Budget, or at All – Tech + Hckr News

06-05-2018 14:02

[…] Why Development Teams Struggle to Deliver on Time, on Budget, or at All 3 by encorekt | 0 comments on Hacker News. […]

Why Development Teams Struggle to Deliver on Time, on Budget, or at All – The XBuzz

06-05-2018 14:03

[…] Why Development Teams Struggle to Deliver on Time, on Budget, or at All 3 […]

Why Development Teams Struggle to Deliver on Time, on Budget, or at All

06-05-2018 14:04

[…] Why Development Teams Struggle to Deliver on Time, on Budget, or at All 3 by encorekt | 0 comments on Hacker News. […]

New top story on Hacker News: Why Development Teams Struggle to Deliver on Time, on Budget, or at All – Latest news

06-05-2018 14:05

[…] Why Development Teams Struggle to Deliver on Time, on Budget, or at All 3 by encorekt | 0 comments on Hacker News. […]

toma

06-05-2018 14:34

it's your problem, your team,, not in general

Tim

06-05-2018 16:06

"But, we rarely do this when it comes to planning. Why?" At every place I've ever worked, it's because managers don't want to hear accurate estimates, which in turn is because their bosses don't want to believe software is as expensive as it is, which in turn is because consumers don't want to pay what software costs. The median price for the app on everyone's phone these days is $0. So if you want to pay someone $100,000/year to write it, they're going to want to cut corners as much as possible, from the top down. Whenever I've submitted an accurate estimate, my manager comes to me and says "You don't REALLY need that long to do this task, do you? It looks like it should take you 1/2 that long. Let's put something shorter on the schedule." (They phrase it like a question or a suggestion, but it never is.) Sometimes, the team meets the shorter goal. Sometimes, it doesn't. Speed is more important than repeatability, though, or correctness. This approach costs more in the long run, of course, but my manager will be promoted out of the way long before those savings will be realized. The only people losing out are the users, but their primary goal is for the software to cost $0, so if it's late and buggy, that's not really a problem. They'll gripe about it to their friends, but not enough to want to pay money for it to be any better.

The Secret Reason that Development Teams Struggle to Deliver On Time, On Budget, or On Scope » Jon Anhold

06-05-2018 17:58

[…] Source: The Secret Reason that Development Teams Struggle to Deliver On Time, On Budget, or On Scope – 7pa… […]

Matt

06-05-2018 19:37

Bollocks, I'm autistic and all system 2 and I still suck at estimating. The real reason? We can't predict the future. Why do we laugh at fortune tellers but expect developers to pull the same stunt off with pinpoint accuracy? Because it suits us to do so, that's why.

Matt

06-05-2018 19:37

Bollocks, I'm autistic and all system 2 and I still suck at estimating. The real reason? We can't predict the future. Why do we laugh at fortune tellers but expect developers to pull the same stunt off with pinpoint accuracy? Because it suits us to do so, that's why.

Marcel

06-06-2018 01:46

Why not adopt agile practices to manage the uncertainties?

Peter

06-06-2018 15:22

You cant easily estimate big projects

Phil

06-21-2018 09:36

Tim, i agree with you totally. People dont want to hear the truth about planning. They want it quicker than it can really be done but for the same or lower cost. I/We see all over with people looking to choose a house builder. The one who plans it all out and costs it fully often doesn't get the job but the one who comes in cheaper and says he'll get it done in a set time does. and often he cuts corners to get it finished which creates it's own problems. You cant plan for every eventuality BUT with experience we can make more accurate plans. But how often are clients willing to pay for that!

What Escape Rooms Can Teach Us About Incentives for Software Development Teams – 7pace Blog

07-31-2018 19:57

[…] Timetracking and historical data can ground estimates, goals, and incentives in the real world. […]

Send