Project Management Stack Exchange is a question and answer site for project managers. It only takes a minute to sign up.
Sign up to join this community
I'd actually like to devide the question in the title in two related sub questions:
In your experience, how reliable is the velocity in letting the product owners forecast the delivery date of a product?
Do you find that the estimates in story points (provided by the dev team) are sufficiently accurate to plan how much can be put in the sprint?
How reliable is velocity?
To give an analogy, it's just as reliable as the weather.
Yesterday's weather, for example, is a Scrum pattern that helps teams quickly calculate how many story points they will likely manage to get done in the next Sprint. It relies on the idea that if you have a stable team and a stable working environment, what you will manage to finish this sprint will most likely be something like what you did last sprint (usually an average of the last 3 sprints is used). If you average, say 100 story points each Sprint, it's safe to say that you will pretty much do something similar next Sprint also. If the team all of a sudden decides that they will do 200 story points this sprint, all other things remaining the same, you can safely bet they will fail.
Just like the weather though, you can only make this prediction on the short term, not the long term. With the weather you can predict with certain accuracy what the weather will be like tomorrow, or this week, but you cannot predict the weather six months from now because a lot of things can change between now and then, you don't have enough information to actually measure things out, etc. And it's the same with predicting delivery dates for software.
In Scrum you work from a Backlog. This Backlog gets continuously refined as new information is learned about the product, as feedback is received, as market conditions change, etc. For this reason you work on the top of the backlog, i.e. the most important things first. The top of the backlog is also more detailed and better estimated than the rest of the backlog. So if you want to estimate delivery dates on the full product, you need to have fairly stable teams, stable working conditions, market conditions remaining the same, no new information discovered or feedback received that could change the rules of the game, and detailed estimates for all the items in the backlog. When these remain stable, you can divide the total number of story points in the backlog to the team's velocity and figure out, on average, how many sprints you might need, then knowing a sprint is (often) two weeks, you can determine a delivery date for your product. In theory.
In practice thou, as I mentioned above, a lot of things change. Your team capacity might change (people get sick, leave, join, etc), market conditions might change, someone discovers some great opportunity and adds to the backlog, some items become redundant, etc. Every time these things change, your calculations involving velocity need to be redone because the target product changes. The velocity itself can also change as the team becomes better at their work, or new team members are added, or older ones leave, etc.
So, although velocity can be a good indicator on how many sprints you might need to finish all of your backlog items (which define the full product, if you like to think about it like that), it doesn't really guarantee you anything. Just like the weather. You have a good confidence level for predicting the weather in the next few days, but not for the next few months.
In your experience, how reliable is the velocity in letting the product owners forecast the delivery date of a product?
Not very.
"Delivery date" is a vague concept. What, exactly, does it mean? Does it mean the delivery of the last item known to be necessary? Or does it mean the next releasable software increment? Some frameworks, like Scrum, call for an increment to be releasable no later than the end of every iteration. Others favor working software, but without necessarily needing to be releasable to downstream processes every iteration.
If you have a potentially releasable increment after every iteration, you don't need velocity to tell you when you'd be able to release something, but it may not have everything desired by your customer or user. You will have a potentially releasable increment, hopefully with some visible changes, at least at your iteration cadence.
If you don't have a potentially releasable increment after every iteration, it depends on what is necessary for it to be releasable. If there are certain items in the backlog that are required and everything is estimated through the last one required for the next delivery, then you may be able to forecast using velocity. However, as velocity is highly sensitive to change in way of working, capacity, and understanding of the product and the domain, it can't be used to forecast very far into the future. The backlog is also not static - items can be added or removed as the team learns. Any change to the backlog can disrupt any possible forecast.
You can try applying burndown and burnup charts to get a rough idea, but I would suggest that anything more than a couple (~2) iterations out is highly questionable.
Do you find that the estimates in story points (provided by the dev team) are sufficiently accurate to plan how much can be put in the sprint?
It depends on how good the team is at estimating. If a team chooses to estimate and is good at it, then yes, it can be helpful to determine a Sprint size. However, there is some evidence that (given sufficient data points), simply counting the number of work items is just as effective as estimating velocity in Story Points or estimating size in hours and may save time by focusing discussions on understanding the work rather than trying to estimate its size or complexity.
Reliability and validity of estimates depend on what you define as an estimate. I am answering this question not from a Scrum perspective but rather generally regarding estimation. I unequivocally disagree with any "no estimate" point of view.
The reliability and validity of an estimate will never be 100% perfect because it is a prediction of the future and we can only predict the future with various indicators, signals, or models and they are interpreted or built by imperfect humans. However, they can and should have a rather high degree of both reliability and validity using sound methods and models.
First, define an estimate. Are we talking about a discreet date to finish or are we talking about the probabilistic range of finish dates, i.e., best case, most likely, worst case? An estimate should be a range of results while a planning value should be a discreet date. The latter is by its very nature very unlikely to meet. The range, however, is extremely likely to be met if the range was established using sound methods.
If you consider your drive from your home to the airport you use, if you made this commute quite a bit, there is no doubt in my mind that you will be able to establish your best case, most likely, and worst case estimate to make that drive. From that perspective, the reliability and validity of estimate is extremely high. For me, my best case is 0.75 hr, most likely is 1 hr, and worst case is 2+ hrs. This estimate is extremely precise and accurate from 6am to about 3pm. I do not have a decent estimate past 3pm because I rarely have made that trip. If I were to choose a planning value within that range, I'd likely leave my house about an 1 to 1:10 before I need to get there. The likelihood I'll get to the airport in that duration is extremely low but, doing that commute 100 times, I'll be on time or early some 60% to 70% of the time.
I am not a Scrum practitioner but I can't see why this same logic would not apply to estimating points or whatever it is gets estimated.
I agree. "Velocity" is a metric that of course has no absolute standard.
However: "within your project, and perhaps within your organization over time," things like these can maybe begin to acquire relative meaning, if you apply [more or less ...] consistent standards to a succession of projects.
In your experience, how reliable is the velocity in letting the product owners forecast the delivery date of a product?
Highly unreliable. This is as a result of:
Do you find that the estimates in story points (provided by the dev team) are sufficiently accurate to plan how much can be put in the sprint?
Yes, in the right circumstances:
It all depends on the familiarity that you have with the team, the familiarity the team has with the problem domain, the familiarity they have with the technology, and the time they have been working on the project!
Complexity-based estimating is an incredibly rich tool for estimation, but it needs to be calibrated. If your team is well-known, and already has good experience with the problem domain and related technology, then velocity estimates based on complexity points will give you a good means to accurately forecast productivity and delivery within a small number of iterations.
If the team is unknown, and/or the problem domain or implementation technology is new, then the ability of the team to consistently judge relative complexity starts to fall.
In that case, you will need more iterations to calibrate your predictions.
If you have a completely new team, you will actually need to regularly re-do the complexity point estimations with each sprint until they stabilize and a level of experience is reached within the team!
The key is that relative complexity is something that folks are inherently good at determining, and when combined with a good iterative process, you can very quickly and easily get a good read on the real rate of progress of the team.
TL;DR - The less well known the team and the less experienced they are with the domain and implementation technology, the more iterations it will take to get an accurate velocity measurement that can be used to make predictions.
You can't make estimates more precise by changing units to Story Points. When it comes to deliverables it doesn't matter in which units you estimate - you still want to predict the date. The idea of Story Points is not to predict precisely, it's the opposite - they imply low precision of estimates.
The best solution here is not to estimate at all. When business receives regular releases with new functionality they simply see that there's progress and that's enough for them to sleep well.
Of course depending on your company there might be red tape and old-fashioned managers who demand to know the dates. But estimating is art, not science. There's no simple way of estimating accurately.
Velocity is stable only in theory, in real life it changes from sprint to sprint and the changes could be drastic. Hence Velocity won't help either.
