5

In our company, we are moving some teams to scrum. Before scrum, it seems we don't have a lot of visibility into the productivity of the teams, and we don't necessarily have a way to measure the quality of the output the teams produce.

With the teams which have moved to scrum, senior management and the teams who have made the switch see the value. The development teams love the collective ownership and protection from disruptions, and the senior managers love being able to have a platform to have their feedback heard in the sprint review meetings. We have also had "anecdotal evidence" in the form of customers saying things like "Wow! What happened to you all? Did you get funding or something?" in response to the perceived faster development.

So, scrum feels faster. It feels nicer. But unfortunately many people still want to see hard data and don't like fuzzy warm feelings or evidence based on observation alone.

Is there any way to measure productivity against an old system where you previously had little to no data? How does one approach convincing other teams in the organization that there are benefits to using agile methodologies like scrum when there's no hard data?

Improve this question
3
  • What metrics do you use to measure productivity in the old system?
    – WBW
    Commented Mar 8, 2016 at 19:59
  • 1
    I wouldn't use hard data in this case. Do other teams deliver software with desired quality and in timely manner? If yes, why change? If not, who pays for it? Bob? Just tell Bob he might save some money.
    – Bartek Kobyłecki
    Commented Mar 8, 2016 at 21:43
  • Scrum won't work well when there's no sense of urgency for a change and it's sponsorship.
    – Bartek Kobyłecki
    Commented Mar 9, 2016 at 8:36

6 Answers 6

Reset to default
5

I worked with one organisation that was in a similar position. They had starting adopting Scrum before measuring the effectiveness of the old way of working.

They did find a few metrics to compare using historical data:

  • They looked at old project documents/emails and measured how long it was from the initial work request to something going in to production, then compared this with the equivalent times on Scrum-based projects.
  • They had some old business user surveys that asked questions like "Are you satisfied with the time it takes to complete projects". Using very similar questions they surveyed the same people again and compared then with now.
  • They looked at historical numbers of production defects for projects and compared them with the numbers occurring on recent Scrum projects.
Improve this answer
3
  • I recon it would be also good to see the time to solve the defects.
    – Zsolt
    Commented Mar 9, 2016 at 8:00
  • I find bullet point #1 to be very helpful. While we don't always document defects, there will likely be email threads that could theoretically be helpful. But getting the data could be incredibly time consuming.
    – jmort253
    Commented Mar 9, 2016 at 14:14
  • The way I saw it done there was not much time spent on it. Several people got together and listed on a white board the projects they had worked on over the last few years (before agile was introduced). They then did their best to remember when they started and when they went in to production. Where people weren't too sure there was some checking of project documents and emails, but it wasn't necessary for a lot of high profile projects that people remembered well.
    – Barnaby Golden
    Commented Mar 9, 2016 at 14:50
2

It may not surprise you that, absent of data from past projects, it's hard to show much comparison. You can use some relative estimation on past projects and take two of similar size or similar functionality and compare them.

You specifically mention the fact that it feels faster and sometimes it is, but that can be harder to get exact numbers on. However, you probably have these numbers: compare value/revenue generation of a project in scrum that is using iterative releases with one that has one big release at the end. If you graph it, it probably looks something like this:

enter image description here

This isn't about Scrum exactly - more about iterative releases or end-of-project releases. It might provide some of that hard data you're looking for though.

Past that, you can also baseline your data right as your team adopts Scrum and then show it improve over time and the team is able to continuously improve. Again, this is more of a promotion for the feedback loops you get in Scrum than Scrum itself, but that's probably ok.

Improve this answer
2
  • Why is that not Scrum? Does scrum say don'tdo it? Focusing on value delivered sounds very Scrummie to me ☺
    – MrHinsh - Martin Hinshelwood
    Commented Mar 8, 2016 at 18:46
  • 2
    It definitely happens in Scrum, but you can deliver incrementally without Scrum too.
    – Daniel
    Commented Mar 8, 2016 at 22:01
1

SHORT ANSWER

  1. Evaluate your previous processes delivery times and schedules (delivery cadence)
  2. Remember, the primary measure of success is working software - show your delivery cadence in Scrum and the difference between now and in the past.

Additional example: I worked on a project once that had 8 tools developed for searching different types of inventory. Each inventory type (50ish total), had its own set of requirements. Under waterfall, my client had 8 tools delivered in roughly 8 years (in a legacy, cold fusion environment). We developed a templating engine in java and built the new tools using Scrum as well as DevOps and built 45 tools in less than 6 months. We used data access standardization techniques such as service-layer API's to be able to predict how data would be stored in the future, which allowed us to essentially write small configuration files for each inventory type - the templating engine did the rest.

Improve this answer
1
  • Hmm, that is possibly a good argument. Because "working software" definitely wasn't always released in timely manner using the old model.
    – jmort253
    Commented Mar 9, 2016 at 14:18
1

In Scrum productivity is measured in terms of actual delivery.

If Scrum is used for Software Development, then software that is valuable to the end users is delivered after each Sprint. An accurate forecast for Sprint Backlog, meeting a Sprint Goal and satisfied Product Owner is enough to measure Productivity.

You can apply objective Software Engineering Measurements, like Function Points, against the software that was developed using other approach and compare it to Function Points delivered in the same amount of time in Scrum.

However, I doubt that this kind of approach would help you convincing Developers to use Scrum.

Improve this answer
2
  • Quick comment here - not sure if you were replying with the "strict Scrum hat on" (forgive me if you were!), but I think most agilists are quickly moving away from velocity and quality as key metrics, and focusing more on actual, measurable value delivered. That is, a PO-dependent team delivering 100 bug-free story points that are hardly used by customers is not as "productive" or effective as a team delivering only 50 story points of well-used features with minor bugs, on a path to becoming domain experts themselves.
    – Jeff Lindsey
    Commented Mar 9, 2016 at 14:16
  • @JeffLindsey While replying, I had a Scrum hat on :) I encourage people to use Scrum when they really mean Scrum described in the Scrum Guide. This sentence: "An accurate forecast for Sprint Backlog, meeting a Sprint Goal and satisfied Product Owner is enough to measure Productivity." means to me exactly what you wrote in your comment, that is, well-used features by end users.
    – Bartek Kobyłecki
    Commented Mar 9, 2016 at 15:43
1

In moving forward with this, keep in mind you usually "get what you measure". Velocity and quality are good, but not at the expense of other areas - happiness, ownership, autonomy, domain expertise/T-shapeness, skill and knowledge decentralization, sustainability, and so on.

In my experience these often have a far larger impact to actual value delivered, are much harder to measure, and will not be represented in velocity or defect count data.

Improve this answer
0

I really like Barnaby's answer, however I'm not sure if you can do the proper comparison that can convince skeptics and data nerds. The problem is with this particular setup that it is not possible to know why the later projects are doing better (or worse).

In my opinion two factors work here: the old way laid down good fundamentals so the new way just plays along with it and builds on it, or the new way actually managed to reform. I believe there is no way to prove the second factor, until one cannot falsify the first one.

It is a common problem in sports. There are athletes who didn't win anything relevant in their career as young athlete, but when they join the adult competition they start winning. Coaches tend to say that the change did the trick, however in several cases the coaches who worked with them when they were younger laid down all the fundamentals (mostly physical) which was key to start winning.

If you really want to prove anything here, go back to the old way for a couple of sprints / iterations. You have data from your scrum, and you can collect data from you old way of doing business, because now you know what you are looking for. Your case must be reversible; if your test shows that the old way is worst than scrum, you have your results. However, if the results are not definitive, the only thing you can say is that the new way feels better, which is a good, but will never be enough.

Improve this answer
2
  • I like this approach, and if we could easily move back to the old model as a "test", I would. Moving to scrum took a lot of effort and time, and we did have to slow down a bit before things started to, seemingly, feel to speed up faster. Even so, when change occurs, sometimes so do people. At Zappos, when the CEO moved the team to a flat organizational structure, 20% or some number of employees left. Unfortunately, it's not possible for us to revert without hurting the organization. Still, I find your explanation of the cause of the problem helpful and will ponder.
    – jmort253
    Commented Mar 9, 2016 at 14:20
  • I do not like this idea, it's bad. For one, people 'know' that they are doing a test, therefore you are at risk of a potentially strong bias. This is a common error in statistical analysis of studies/experiments. If you know you are being testing or experimented on, you perform your functions and actions differently than in a normal setting. The results would not be scientific.
    – Jon Luzader
    Commented Apr 12, 2016 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.