We're getting started with Scrum and have had a few debates about whether we should estimate in story points or ideal hours. The team is pretty divided, so I'd like to know if any real studies have been done about which is more effective. I've certainly read a lot of opinions on which is better and, while there are good arguments for each, I haven't found a single book, article, or post on a web page that's backed up by concrete numbers.

So, have there been any studies done that I haven't come across? How about psychology research on absolute vs. relative estimates? If those don't exist, what about real-world examples of what has worked for different teams in various types of companies?

up vote 7 down vote accepted

Mike Cohn in Agile Estimating and Planning describes estimation using story points vs ideal days in Chapters 4-5, then compares them in Chapter 8. Regarding research:

There is credible evidence that we are better at estimating “this is like that” than we are at estimating the absolute size of things (Lederer 1998; Vicinanza 1991).

The sources of these citations are given as

In general, ideal days may be easier to comprehend for some people / teams, however the risk is that it is too easily interpreted (even unconsciously) as concrete days (elapsed time), which then may even be viewed as a commitment (by e.g. management). This is mainly why story points are used by many, to avoid any chance of misinterpretation. With story points it is always clear that we are estimating relative size, not duration.

It may also be important to note that story points can not be directly translated into effective task hours / days; over a longer time period, a team of course is able to calculate how much time a story point may take to complete on average; however that does not mean that you can use that value to calculate the time required to complete a specific item. In real life, a concrete item of 2 story points may turn out to require the same amount of time to complete as another item of 1 story point - or it may require 3 times as much time. However, in a larger sample these individual fluctuations tend to cancel each other out nicely, hence story points can be used to estimate team velocity in the long term.

Mike himself clearly favours story points:

My preference is for story points. I find that the benefits they offer as pure measure of size are compelling. That story points help promote cross-functional team behavior is a huge advantage. Shifting a team’s thinking from “my part will take three ideal days and your part will take two ideal days so the total is five ideal days” is very different from “overall this story seems about the same size as that one so let’s call it five story points also.” That a story point to me can be the same as a story point to you, while the same may not be true of an ideal day, is another big benefit. Two developers of different skill or experience can agree on the size of something while disagreeing about how long it will take to do.

The shortcomings of story points are indeed short. Yes, it’s easier to get started with ideal days. However, the discomfort of working with nebulous story points is shortlived. Ideal days are definitely easier to explain to those outside the team but we probably shouldn’t choose based on how hard it will be to explain to outsiders. That ideal days are so easily understandable causes problems as well. In some organizations there will be pressure to make an actual day closer to an ideal day. Pressure for more focus and concentration on our work is fine. But organizational pressure for each actual day to be close to an ideal day will also have the effect of causing us to estimate in actual time while calling it an ideal day. That is, an ideal day will become redefined as “a day where I make six hours of progress and do other things for two hours.”

Update

Just bumped into another article from Jeff Sutherland's blog with research details and links:

Story Points: Why are they better than hours?

  • This link seems to work: www.idi.ntnu.no/grupper/su/publ/ebse/RK15-reviewexpertestim-jorgensen-jss04.pdf – James Kingsbery Feb 26 at 22:33

Story Points: Why are they better than hours?

Take a look at this blog post by Jeff Sutherland on Story Points: Why are they better than hours?

This is further vigorously defended by him in the discussion below the blog.

In this blog post he cites the following examples. These may be helpful to you as "real-world examples of what has worked for different teams in various types of companies":

  • A study of 80 multimillion dollar projects at GSI Commerce (now owned by eBay) showed that the best experts in the company were totally incapable of estimating how much time a project would take by the people who actually implemented it.

  • Rand Corporation research in the 1940's showed clearly that humans are not good at estimating hours and practical experience repeatedly confirms the research. The recommended Delphi approach to estimation which was adopted in software development as the Wide Band Delphi technique. The same technique is now embedded in the practice called Planning Poker for agile teams.

  • A CMMI Level 5 company determined that story point estimation cuts estimation time by 80% allowing teams to do more estimation and tracking than a typical waterfall team.

  • A telecom company noticed that estimated story points with planning poker was 48 times faster than waterfall estimation practices in the company and gave as good or better estimates.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.