Project Management Stack Exchange is a question and answer site for project managers. It only takes a minute to sign up.
Sign up to join this community
This project--developing web-based modules--is currently in its testing phase. On a daily basis, the team captures testing results and progress using a cumulative flow diagram. The testing phase is only beginning and is scheduled to go out until mid-August. Based on the data exhibited here, what are your interpretation and findings on the health of this phase so far? What risks does this chart exhibit?
All right, don't have all the information, so making some assumptions here...
The goal of Cumulative Flow is consistency. Consistent cycle time from when a task is started to when it is finished is the goal. You measure cycle time horizontally from left to right.
At the start of your grid you had a cycle time of a couple of days (a task started on 5/28 was closed by 5/31. But by 6/2 your cycle time has grown to at least eight days before the cycle goes off the grid, and by a back of the napkin progression I'd say it's a good two weeks (draw a straight line from the left edge of proposed on 6/1 and see that it does not intersect closed). So from when a bug is proposed to when it is closed is growing. Troubling is the largest growth is in resolved.
When resolved cycle time is growing, in a traditional development, that usually tells me that the development fixes don't work the first time. QA finds a bug, reports it, dev fixes it, fix doesn't work, goes back to dev, tested again.
This is a classic trap in traditional development. The longer you go from when a bug was introduced, not found but when the developer introduces it, the harder it is to get it fixed. The developer has already moved on.
So short answer of what I see. Developers will still be fixing a huge mass of bugs when they are supposed to be "done" and probably already working on the next release. Continue this cycle for three or four development cycles (typically years in large companies) and you will end up missing an entire release because you have to spend the time to fix all the technical debt.
Personally I don't like cumulative charts for defects etc., I guess my brain does not process them easily. I prefer to see defects raised/closed per day etc. so you can see the curve.
Having said that, this appears to be a "normal" curve for defects being raised over time, with the closure rate tracking the open rate quite closely.
What would worry me, if this were my project, would be the missing information (on a separate table/graph) that shows test case progression over time and for that one I do love cumulative actuals against cumulative planned. The worry being: Is the raise rate tailing off because of slower than planned progress through the testing? It is impossible to forecast (or to be more accurate "guess"), from numbers like the above, what the eventual level of defects found will be or whether the current open/close rate is likely to meet the acceptance criteria at close of testing, because we can't see how much testing has been done and how much remains.
I would personally also like to see the numbers for "blocked" test cases as this can store up real schedule problems.
