I cannot speak to your particular situation but yes, it is pretty common that not everything is done.
This is in fact a logical consequence of the structure of the team, and you being a developer are in a prime position to appreciate it.
Load and delay
If I were to ask you “do you run a server at 100% load?” you would likely laugh in my face. So I might follow up with “OK if that is ridiculous, what sort of load is acceptable?” and you might offer a number of responses including simply “how do you define load?”
But I think we could settle on the idea that more or less no matter what the definitions are, you would not like server loads above, say, 10-25% of capacity: certainly not much above 50%. And were I to ask you “why not?” you would very likely first give the Wrong Answer: that you’re worried about hardware failure. That’s a fair point but usually if you’re doing it right you’re already guarding against that risk. And then second you would give the Right Answer: that you’re worried about latency spikes. If the server doesn’t have spare capacity, it can’t begin a new request until it is done finishing up old ones. And that’s true of people, too.
The core problem is that a sprint model typically assumes one way or another that there is 100% developer load, and developers respond by padding out their estimates with safety buffer so that they can hopefully only operate under 50% load or so. Then you have natural problems dealing with the same two problems: hardware failure (employee burnout) and latency spikes (hey, Legal said one week ago that Floozels must only be awarded for the Flotsam that is also Jetsam—why isn’t this fixed now?!). In practice nobody wants to accept the latency spikes and so they simply throw story points of developer time out of the current sprint in order to keep latency low with their bosses and the business at large. Similarly any “some other issues came up” workloads appearing in stand-up have the same origin. So your developers are taking care of their own health and that means some amount of doing non-work, and this sometimes bleeds into losing story points; and your developers are doing one-off requests or unexpected workloads, and that sometimes bleeds into losing story points. Those are the two main sources.
In fact the problem is somewhat larger than this because safety factor in estimates is much much bigger than even the typical advice to “always multiply estimates by N” suggests. The reason is that the private estimate for completion completes the goal with, say, a 90-95% probability, which for realistic situations might already add a factor of 2 of safety versus the 50/50 chance of completion that we should idealistically be able to target; furthermore a PM then buffers those estimates with some additional padding anywhere from 20-100% when communicating with the business as a whole. Take any software project, and something like three-quarters or more is safety buffer. We devs would be completely scandalized at the idea that any project would ever run over-time or over-budget if we understood how much safety buffer we have to spare to complete it. The typical excuses we give, “oh we will be late because in our last week a hurricane took out our data center causing all of our systems to be down for several days, but the team worked together and pushed really hard and we got everything working and we’re only going to be late by three days” are laughable simply because given the safety buffer we should have delivered way before that hurricane hit.
Waste and purpose
So there is a paradox of we have huge safety buffers and we still deliver late. The solution to the paradox must be waste; we must be wasting all of that precious safety buffer. This happens in many different forms and enumerating all of them would make this answer very long. However I think we need to point the finger first squarely at management simply because the developers lied in the first place; they padded their estimates first from 50% to 95% and then they added another factor-of-two on top of that, and this means that management has not created a space where they feel safe in being totally honest with their estimates. Why do I feel free to take care of myself by answering a question on Project Management SE? Because I am at the start of a sprint and everything has luxurious unspent safety buffer still and so nothing seems desperately urgent. Come back to me on the last three days of the sprint and see if I am still on PMSE, heh.
I can also give a quick sort of aspirational vision for how to solve it. In theory there is a dependency graph of tasks-to-be-done for your Sprint Goal or any other release you have, and it typically involves some real stinkers that nobody wants to take on, like “figure out why the QA server is crashing even though we have implemented workarounds to make it not crash so that QA is unblocked, we have no idea what’s causing it, good luck, don’t break anything.”
If you think about commits on this sprint to your develop or master branch, you see that even though the dependencies are in many cases not enough to structure the thing completely, the tasks will ultimately form a directed acyclic graph and there is some critical path from sprint-start to sprint completion: the path through this graph of work-as-done which takes the longest amount of time. Then there are several side-paths—other devs working on important things which are not on this critical path—which have to merge into that critical path. I don't care about them unless they delay the critical path, really: people can be free to goof off within some reason on their side-paths.
In the ideal project there is therefore at any given time exactly one developer who is in a state of panic. They are working on the critical path, and they know that this responsibility is upon them. The critical path is something of a baton or hot potato that they are looking to pass off to the next person as soon as they can; and they are given considerable power to loop in other developers to help them with their task, or to compromise scope of their feature to be good enough to pass the baton on. PMs and other facilitators have several roles:
- They encourage people. Deadlines in this model are routinely missed because they only have a 50/50 chance of being completed, so facilitators need to remind people that this is okay and we have a project safety buffer precisely for this; this thing is still urgent because we’re now digging into the project safety buffer but we have it so that we can use it when this happens.
- They prevent burnout by ensuring that the baton keeps being passed from person to person (as others are given a great deal more latitude to rest).
They coordinate dynamical changes to the critical path. Suppose the schedule looks like this:
Time: 0 1 2 3 4 5 6 7 8
Side-Path: [Alice-----][Bob-------][ safety buffer ]
\
Critical Path: [Carol-----------------------------------][Dan----...]
in this diagram, Alice and Bob are on a side-path; they both have a 50/50 shot of completing in 2 units of time, Bob’s work can’t start until Alice’s is merged. They share a safety buffer of 3 units of time for their thing before it merges into the critical path; it is needed when Carol hands off her changes to Dan; she has a 50/50 shot of completing in 7 units of time. Now when this schedule meets reality, you have to accept lots of “it’ll be done when it’s done” and trust these four to be doing their best; Carol is aware that the entire project right now is riding on her work and is in the state of panic. But what can happen is that Alice was pulled into something else and it is t=3 and she has at least one time unit before she can finish and hand it on to Bob at t=4; but Carol is estimating that her task was much easier than expected and is going to be handed on to Dan at t=5 or t=6. So in theory facilitators should be noticing when the safety buffers on these side projects are running out or when the tasks on the critical path are getting done early (either alone is enough to potentially lead to a shifting of the critical path) and letting folks like Carol know that they’re no longer custodians of the hot-potato: that it has shifted and they can relax a little because now the person who is most affecting our precious safety buffer and hence our ability to meet the deadline is Bob, who now needs to feel that responsibility.
- They lubricate these jumps. When Carol (or Bob...) hands the baton to Dan above, the facilitator is ultimately asking Dan to completely drop everything that he is doing: this will be disastrous if he does not have advance warning. So they are asking the panicking Carol for realistic progress updates, but again they do not care if she is at her deadline or not. Remember, that deadline was only estimated at being 50% chance of completion; it is therefore rightly meaningless for holding anyone accountable and there is no shame in missing it. So Carol needs to be approached with gentleness and good faith and asked how soon she thinks she can be done with her task, purely so that the facilitator can tell Dan, “hey in around two days or so I am going to ask you to drop everything else and work on this one specific task, you will have the baton, please clear your schedule for that and start getting ready to sprint with it.”
If facilitators can do all of that with compassion then they can build a trust and hypothetically deliver projects significantly faster, because the key feature here is that we are treating the safety buffer as a precious resource. Teammates are under pressure proportionate to their ability to affect that buffer: the ones who most affect it are therefore not wasting it with “oh let me go check Reddit...”. The others are allowed to rest so that you get the benefits of a well-rested team. And finally, we do not waste anything we get when someone delivers early because we communicate.
Note that this is a very pretty vision but I have to attach some caveats. This is not a universal structure of all business. It is a derivation of some general principles for a specific context where every object being produced is a fundamentally new artistic endeavor and only is produced by some big complicated cooperative effort. Your software teams may be working in a different business sector which does not work this way: for example Valve no longer makes big software games to be bought for a big price tag but instead makes skins and user customizations sold on Steam to get a lot of micro-sales; one can imagine that the Valve Before would have strongly benefited from this sort of structure whereas the Valve After probably would not see those sorts of returns. But those sorts of patterns are pretty common in software development so it is reasonable to think that many software shops would benefit from something looking similar to this.
