Why are developers expected to estimate tasks at all?

Question

This management tendency is the worst part of being a developer. Software development is not carpentry. Almost everything a developer writes is unique, they have never built that particular thing before. We are not cabinet makers repeating a variation of something we've built hundreds of times before.

I've been in this trap so many times:

1. Give a very wide estimate with a lot of padding
2. Get pressured to be more accurate and to bring the estimate down
3. Get pressured to work unpaid overtime to meet that estimate
4. Watch management get congratulated by upper management for running a tight ship

In my ideal world, management would simply tell me which task they want me to work on next and then as it proceeds ask for a regular update of what percentage of completion it is at. It would then be the manager's responsibility to extrapolate my likely completion date, learn my accuracy over time, adjust to it and manage expectations of upper management.

If developers have to be the ones carefully calculating estimates and managing expectations then there's really no purpose to middle-management. The developer might as well be speaking directly to the client if the only value added by managers is to pass estimates from one hand to the other and then brow beat developers when things don't work out.

Why are things done this way when there's decades of history showing how badly it works?

Answer 1

You Have an X/Y Problem

Why are developers expected to estimate tasks at all...[when i]n my ideal world, management would simply tell me which task they want me to work on next and then as it proceeds ask for a regular update of what percentage of completion it is at. It would then be the manager's responsibility to extrapolate my likely completion date, learn my accuracy over time, adjust to it and manage expectations of upper management.

Most of your question is really a rant about how things work at your workplace. Discussions about toxic workplace practices per se are out of scope for PMSE.

In addition, your ideal of just plodding along and expecting the rest of the organization to adapt itself to you is unrealistic, and completely unmoored from the purpose of business, which is to make money by providing a product or service within a given schedule, scope, and cost. The company's customers also have goals, deadlines, and budgets that need to be met, so you're basically positing a world where nothing can be predicted and your personal cadence sets the pace for the entire market. That's unreasonable on its face.

That said, estimating poorly or failing to use modern techniques such as batch and queue theory to manage estimates certainly leads to poor results. That's what agile frameworks address: the inaccuracy of estimates that aren't based on small, estimable, and sustainable batches that can be successfully delivered at a predictable cadence. There are even people who advocate for flow and cadence as the estimation tool (using the poorly-named #noestimates hashtag on Twitter), but there must still be bounding boxes around any project that has constraints.

Your X/Y problem is that you have a problem with your company setting deadlines that you can't meet, then asking you to estimate tasks in a way that leads to inaccurate estimates and blame when deadlines are missed. That is "X". You have decided that not having deadlines or estimates is the solution to your problem, and posit this solution "Y" as the answer to the perceived process problem solely as it affects you. Unless you shift the frame to include the rest of your team, other parts of the organization, and a business perspective then you are simply venting rather than becoming part of a constructive solution.

Solve for "X", Not for "Y"

Implicit in your X/Y problem is a lack of effective communication with your middle management and senior leadership; possibly even within your team. You need to address the communications gap before you can do anything else.

Aside from your frustration, your team and organization need to address the root cause of unmet expectations. Selling projects without sufficient controls on scope, schedule, and budget is a non-starter, so your team and the company should both understand what the constraints are. It is unlikely that it is your job to define any of these constraints, but it is certainly your responsibility to talk with your team and with line management if you are continuously working on "death march" projects.

It is likely that one or more of the following need to happen:

1. There needs to be more clarity about specifications, deliverables, time frames, and scope.
2. There should be more input from your team leadership about whether the deliverables promised are reasonable within the budget, skills, and resources available to the team.
3. Your team needs to get better at decomposing work into smaller work units that can be estimated with reasonable accuracy.
4. You need to decide if the problem is the company culture or you.
5. In either case, if the process can't be fixed then you need to find a different company that's a better fit for you.

The first few points are things that can be fixed in any organization once a root cause is identified and acknowledged. The last few points are really just workplace advice, which seems necessary because your current framing of the issue doesn't take anyone into account but you. If you are unhappy or a bad fit, your only realistic options are collaborate with your leadership on solving whatever underlying process problems exist or moving on to another role that makes you happier.

Project management is the practice and profession of delivering a finite product or service within the constraints of scope, schedule, budget, and quality. All projects have constraints, and all successfully-managed projects have controls to manage those constraints within acceptable tolerances. While how the constraints are controlled may vary, the need for controlling project constraints will not go away.

Trying to wish project management into the cornfield is not a useful endeavor. Instead, see if you can be part of the solution. If not, brush off your resume and move on.

Answer 2

Frame Challenge

Almost everything a developer writes is unique, they have never built that particular thing before. We are not cabinet makers repeating a variation of something we've built hundreds of times before.

Speaking as a developer and Scrum Master: Yes, we have. We have done parts of the software dozens if not hundreds of times. We are by definition the people best capable of estimating what it is going to take, because we have the best experience in overseeing what is required for that "simple feature request from a non-developer".

This applies especially if you are working agile (you labeled your question scrum), where:

1. User stories are relatively short.
2. Estimates are relative to earlier stories and in arbitrary story points.

Your points 2, 3 and 4 hardly apply if you work agile. At least that is what the Scrum Master and Product Owner are supposed to guard against.

Surely this can fail, but then you are not working agile; you are not providing a safe, reliable, and trustworthy work environment, which is going to cost the developers and the company in the long run. (Actually, this is regardless of methodology. It also applies when you do 'waterfall' development).

There are of course long-term goals, and is the job of the Scrum Master and Product Owner to translate them and to balance the velocity of the development against the goals—which is material for a new question.

Answer 3

Estimates are generally used for managing risk and costs and those things are legitimate concerns of management and everyone else in a commercial enterprise.

Several techniques can make estimation simpler and less contentious. Continuous, iterative and incremental delivery are probably the most important and widely used tactics to control software development cost and risk. If you deliver software features every day or every week then estimating each increment is much simpler and the consequences of any delay or under-estimate are much less significant.

Many development teams are expected to be responsible for their own cadence of delivery and their own estimates precisely because they are usually the people best able to do those things. Regular engagement with stakeholders and prioritization of work in terms of business value and risk are also essential.

Many teams choose to make relative estimates (story points) rather than absolute ones and then measure actual productivity based on evidence rather than guesses.

Check out sites like https://www.agilealliance.org/ if you want to learn more about ways of working in software development.

Answer 4

Why are developers expected to estimate tasks?

In Scrum, we don't have "tasks". We have stories. If you're estimating tasks, you might be making a mistake.

Why are developers expected to estimate stories?

This is a question of two parts:

1. Why must stories be estimated?
2. Why should developers be the estimators?

Why must stories be estimated?

Estimates are essential for sprint planning. Without them, it's hard to know how much work could be achieved in a sprint unless each story is exactly the same size (and making stories equally-sized is itself a form of estimation). Each developer's perception of whether the sprint goal could be reached will depend on their own internal assessment of the sum of all the stories.

With agreed estimates of individual stories and an estimate of the team's capacity, choosing what to do can be approached more objectively (at least for the first cut; there's still a need to look at the whole and see if it makes sense).

Why should developers be the estimators?

Some (non-Scrum) practices have estimation done outside the team by specialised estimators. This doesn't work very will for an agile team, for several reasons:

• Waiting for estimation can block a story from being Ready to begin implementation. Developers are in a position to estimate the story right when the estimate is needed.
• Developers have the institutional memory of implementing similar stories in the past, so are more likely to anticipate the problems that are likely to occur, or the extra activities not specifically mentioned in the story ticket.
• The discussion needed for a consensus estimate exposes deficiencies in the story description and ensures that they are resolved before work begins (e.g. the need for infrastructure changes to support the development, or a need to gather user work-flow information to guide a UI design).

What is an estimate?

The specific case you mention is a management anti-pattern:

1. Give a very wide estimate with a lot of padding
2. Get pressured to be more accurate and to bring the estimate down
3. Get pressured to work unpaid overtime to meet that estimate
4. Watch management get congratulated by upper management for running a tight ship

One reason that we use abstract story points rather than time estimates is to avoid this culture that sees the points as predictions or forecasts rather than what they are: estimates. Another reason is that estimates can include points for complexity and for risk, rather than being based simply on time expended. This is covered well in Why use story points instead of hours for estimating?

Abstract points are naturally self-adjusting. If we have consistency in what size story counts as "1 point", then it doesn't matter if that's not the same as a different team's 1-point story, as our measure of capacity is measured in "our points" and their capacity is measured in "their points". If every story in our time is "padded" to give a margin of safety, then our velocity measure will reflect that; conversely, if every story is shaved down to try to fit it into a sprint, then the velocity will reflect that instead.

There's certainly a real danger in working extra time to achieve the sprint - now our velocity, and hence future capacity, takes account of that practice, and the overtime will become required in every sprint. And that's not a sustainable work practice. Much better to fail to achieve, and measure what was actually completed, in order for velocity to represent a reasonable expectation of what's a likely capacity in future sprints.

How can we correct perceptions?

A large part of the question seems to be predicated on management misconstruing what estimates mean. When communicating with managers, it's very easy to be misunderstood - this is often the interface where the agile world of responding to change meets the business/financial world of having medium-long term plans for making promises to customers.

In such communication, we need to regularly reinforce what an estimate means. Story estimates are by developers and for developers. As an open and transparent team, we're happy to share how we work, and show how we use our estimates to say, "we can do A or B this sprint, but not both." But we need to be clear that our commitment is to the Sprint Goal, not to individual stories, and that our estimates are quite rough categorisations (half of the time, effort will be more than anticipated and half the time it will be less - this becomes useful when stories are grouped into a sprint, because the law of averages means the aggregate total is likely to be close to the sum of the estimates).

When I talk to managers, I'm always aware of their desire for predictability, so I'm always careful to distinguish between the Sprint Goal and the stories that contribute to it, and explain which parts are most likely to be incomplete when the sprint ends. Most of our sprints achieve a large part of the Goal, but often missing some of the functionality. That's why the Sprint Review exists, to assess progress so far, and work out what needs to be added to or removed from the product backlog to get to a releasable product at the right time. Any worthwhile product manager is ready to have conversations about trading scope for timescale during the product development - that's the nature of creating something new and original.

Answer 5

There are many possible reasons for creating estimates, including:

• The process of estimating can help the developers to think through their work
• There may be hard business deadlines ("release the product for Christmas")
• Marketing, sales and other departments may have use of estimates ("let's delay the marketing campaign for product X as it looks like development is running late")
• There may be tradeoffs in planning ("if product X takes 2 months we can look to pivot to product Y, otherwise we should hold off until next year")

Having said that there are good and bad ways to do estimating.

...the only value added by managers is to pass estimates from one hand to the other and then brow beat developers when things don't work out

This is an example of the bad way!

A good estimating process is collaborative and it takes into account uncertainty. A common approach is to use ranges, for example a development team might estimate 2-5 months for a piece of work early on in the project. After a couple of weeks of work they feel confident to narrow that range to 2-3 months, and so on.

Answer 6

Why are developers expected to estimate tasks at all?

Because if we didn't, who would?

Do you want a project manager (who, as you've no doubt experienced by now, knows nothing about the realities of developing software) estimating in advance how long a programming task is going to take? Such that you then have to try to meet that schedule? I've had that happen to me a couple of times, and it ain't pretty. Ill-informed managers really do have a propensity of imagining that writing code is like making cabinets, churning out minor variations of something that's been built hundreds of times before.

Yes, trying to estimate an unknown task is hard, and it's no fun, and there are some pretty bad consequences of getting it wrong, in either direction. I get it: I'm a developer, too. (Normally I hang out over on Stack Overflow. I signed up for an account here on pm.stackexchange just so I could give you this answer, so that you can hopefully take me seriously, even if you're suspicious that all the regulars here are just a bunch of project managers who don't get it, after drinking too much kool-aid.)

The thing is, estimating how long a new task is going to take is a skill you can develop, like any other. If you get good at it (demonstrably good at it, by having your actuals track your estimates pretty well), people will start trusting your estimates. They won't engage in the fallacious task of trying to bargain you down, and when you're faced with a really uncertain task, that you can't give a good estimate for because the task truly is unknown, they'll listen to you and work with you on that basis, too.

But, when you think about it, most tasks are not "truly unknown". They're like tasks you've done before, in one respect or another. In fact, in all but the newest startups, many if not most tasks are incremental modifications to code you've already got.

I pride myself on my estimates, and the people I work with love them, because they are generally accurate. (It helps that I've been doing this for a long time, such that I've got a pretty good intuitive feel for it.) My boss doesn't even multiply my estimates by 2 any more.

I don't know the environment where you work, but I'm going to go out on a limb and say that the other engineers in your organization (the EE's and ME's and whatever) are not designing knock-off cabinets, either. They're designing brand-new widgets, too — after all, if the widgets weren't brand-new, they wouldn't need designing!

I know (believe me, I know) how wonderful it would be if everyone could just go away and let us polish our code to perfection, no matter how long it takes. (That's exactly how I run my personal projects.) But, you know, seriously: no matter how much you hate schedules and deadlines and Gantt charts, a real company (the kind that can pay you money) has to have them! How else can they know whether the project's delivery schedule will meet the expectations of the salesman and accountants — and customers! — who are depending on it? How else will they know how many engineers to assign to the project? How else will they know — if it comes to that — that the project is impossible, and needs to be canceled now, before they waste any more money on it?

Don't say, "But Software is different!" That's a prima donna attitude, that won't win you any friends. Software is fundamentally different in some interesting respects: it's infinitely malleable, its manufacturing cost is near zero, the lead time for raw materials (1's and 0's) is zero, and (because it's infinitely malleable) the requirements are always changing at the last minute. But my advice is to hold the "Software is different!" card close to your chest, and play it at the end, when people are asking for last-minute requirement changes to counteract problems that have cropped up in the other disciplines that aren't so malleable. That way you can be a hero, and get some slack for the fact that the last-minute requirement changes are going to blow out your schedule (again). But don't try to play that card up front: it just makes you look like a whiner.

[Disclaimer for all the regulars here on pm.stackexchange: Apologies for the newbie post, which is written in total ignorance of this community's penchants and proclivities. And I hope no one felt I was saying that you were a kool-aid-drinking project manager who doesn't get the realities of software development, because although I did use some of those words, that's certainly not what I was trying to say.]

Answer 7

Why?

The why, as stated by others, is simply that the top level wants to know "how long" and that has to bubble all the way down to asking the person that's going to do the work.

The frustration "solution"

The actual problem is your frustration, and the answer to that is that you need to partially skip #2. Don't succumb to pressure by just lowering your estimates. With experience, you may get better at estimating (I've been doing this for 30 years, and I'm not sure that I've gotten noticeably better, but I also have no sense of time in the first place) but you don't do yourself any favors by randomly lowering. If you learn as you get into a problem that it can be done faster, fine. But you may also learn that it's going to take longer and need to adjust your estimate up.

Scrum reality

To the others talking about story points and that estimates aren't supposed to be deadlines, the reality is that that isn't how it works in a lot of places. I've never been in a scrum team that actually follows that "by the book". Yeah, we use story points, and they are relative to "other work". Ideally, it's supposed to be that during initial stages, you've found the lowest effort item in the project, assigned it a 1 and everything else is relative to that, or something. But I've never been a part of a team that actually does that step, and people jump around teams or even companies so much that they have no idea where the "1" is at or how much effort it took anyway. So in all of our teams, the defacto baseline is "some imaginary story someone did at some point in the past that took exactly 1 day". So story points are days. On top of that, management one level up wants actual numbers anyway, so we estimate stories with story points and then are required to add "remaining hours" to each task within the story. I multiply story points by 8, subtract the "standard" tasks, like "get code ready for deployment" and whatever is left goes on the coding and testing tasks.

Ultimately, this feeds back into the why. Regardless of what your team or scrum master or product owner think about the estimates, some level up there somewhere wants a deadline.

TLDR

So, long story short, we are asked to estimate because that's who should be estimating. But it's up to you to

1. make sure your estimates are realistic
2. be conscientious about trying to meet that estimate, and
3. to also try to make sure you are on a team that realizes and accepts that actual time may go over or under that estimate.

Often the only way to do #3 is to continue with #1 and #2 until they accept it or fire you, or you find a new job. But adjusting your estimates down for management is never going to get you #3, and every time you work your butt off to meet the new goal, it implies to them that you weren't doing #1 and #2.

Answer 8

This is an experience problem, not a management problem

Software development is not carpentry. Almost everything a developer writes is unique, they have never built that particular thing before.

This part is not actually true, Carpenters and Developers face about the same level of uncertainty with each project. If you've ever tried hiring a young independent carpenter, they will often way under bid the project, and have to come back asking for 2-3 times as much money to cover the unexpected materials and labor, but when you hire a carpenter with 10-30 years of experience, they can estimate a project within a 10% margin of error for actual variance, but know to build in a 50% margin of error for when the scope changes a bit, and they are on budget every time. Not because every cabinet is the same, but because they are all made up of the same basic bits and parts. These same margins of error to experience levels are seen in the software development industry, BUT software developers are on average 11 years less experienced than carpenters so we see a lot more novices and very few masters in the software world.

Those experienced developers who are out there have already learned so many new languages, done so many API integrations, built so many physics models and CRUD interfaces, etc that even if they do have to do something "new" they can break it down into 20 smaller things that they already have done many times before so they know just how long it will take.

Management must expect you to estimate your project

Only developers know how long it takes to write software, just like only carpenters know how long it will take to make a cabinet. Yes, you in your novice ways may only be able to estimate within a 300% margin of error, but if that is how close you can get, imagine how much worse it would be if your manager had to guess? When I ask my project manager how long HE thinks a task should take, if the job is just like a previous one, then he can usually guess pretty close, but if it's not something he's seen before he's can easily be off by 2000%... and no, that is not a hyperbole. He literally does not no how a single part works where even a novice developer will have some foundational knowledge. Even a novice developer knows SOME of the processes and requirements of a new project before he starts, and that still makes a young developer's estimate the best possible guess, when no other estimate can be closer.

Software development is a business. It must either make money or add a value that is worth a certain cost. While your job may be to write the software, it is your manager's job to sell a client on the idea of paying for the software. Even if you are doing inside development, your manager will still need to "sell" the value of your work to his superiors or investors... and no one buys a product without knowing what it will cost. So, if you can't provide an estimate (even a bad one) then there is no business.

As for deadlines, if you can't be held accountable for your estimates, then there is also no business. Most development is a contractual agreement. When you give a manager an estimate, he has to turn that into a bid to sell your time to the client. His job is not just to say, "this is 1000 hours of work". His job is to talk to the client, get a feel for them, and decide if he can get away with asking for $100,000 or $150,000.

He has to consider how much residual income this client is worth. He has to consider if it's worth risking the company's reputation asking for add-on costs if things go over budget, he has to consider if he should offer the client a payment plan, he has to consider dollar per hour how much each person working on the project costs the company. He has to consider how much he has to pay you to NOT work if too many bids fall through. Even if you are doing Agile development, these are still the kinds of numbers he has to discuss with clients. Agile is a method for organizing development, not a substitution for responsible financing.

So, no he's not just getting high-fives for projects taking how long you said they should take, he's getting high-fives for all sorts of responsibilities and skills that you don't even need to think about that have to come together just right to keep the company going. Yes, he needs your expertise where he needs it, but he has a whole other litany of jobs to worry about that you as a developer don't see.

The real problem in the development industry

Going back to the Carpenter problem, Carpenter's solved this issue centuries ago when they developed the guild system of management. In guild work, you have apprentices, journeymen, and masters. A young and inexperienced craftsman should never work without the oversight of a master, and a moderately experience craftsman should only work independently on "basic" things, but there are just too few master developers to go around and no unified system for licensing masters to make sure that they are as good and experienced as they should be.

Some larger firms actually have partially adopted this system though. We generally call them Intro, Junior, and Senior or Tier I, II, & III developers. If you work in such a firm, then the Senior Developer is usually the one who writes the estimates, but the issue is that most of these companies have only half adopted the guild system. A person can be "senior" in title, but there are no quality controls to make sure that he actually has the skills that he should have; so, he may not be qualified to write estimates for apprentices (or even himself for that matter.)

Give the industry another 10-20 years to mature, and you will probably see it get to be almost identical to the carpentry industry... for now though, companies are just doing the best they can with the experience that is available.

Answer 9

Scrum should not estimate in time units, only story points, for all the reasons the other replies mention.

My own rule of thumb, after 30 years as a developer (and also in life outside work), for any medium to large chunk of work (say a complete feature) is "our very best professional, honest, unpadded estimate given all we know right now - then multiply by 3".

Most old hands that I have discussed this with seem to broadly agree, but might go for 2.5 times. 2.5 or 3 x seems to allow for our average levels of optimism and stuff we forget about at the beginning. This is to get stable, robust, well-written, production-ready code.

Don't forget ELAPSED time would always vary (can you get meetings with the POs? How many meetings do you have? Admin? Unexpected urgent bugfixes? Can you get the testers time?)

Answer 10

There is no history of how badly estimates work. Further, how your management or your customers treat your estimates is not a sign of how the estimating process is worthless but a sign of poor management.

Estimates are mandatory, despite the no estimate crowd that seems to be brewing in the software industry. No customer will ever buy something with a blank check.

Finally, you have to estimate in order to measure progress. It's the fourth of the five immutable project management laws. Every estimate or planning value you choose, whether you're doing carpentry or true first of its kind work, that planning values lives in a range of probabilistic results. It's not about getting it 100% right. It's about choosing a planning value that has the appropriate amount of risk and then measuring performance so you can predict the variance you will end up with.

EDIT to answer Comment:

Firstly, true FOAKs are quite exceptional. If it is a FOAK for your team, then you need to find different talent. If that's not possible, then you need to learn how to manage up. If you have a ton of uncertainty around an estimate, then you need to say that to management. If all you feel is pressure to commit to something you cannot confidently take on, and then blame you for failing to meet those deadlines despite what you told them, then put your resume on the street. You work for idiots. But you cannot find something magical about an estimating process if you have that kind of management. Nothing will work. Solve for the right problem.

Answer 11

Honestly? I don't think you can get an answer to this.

Estimation is an inherited practice from other businesses. Most businesses, it's relatively straightforward to estimate. How long will it take you to balance that ledger? About the same amount of time as the previous one, assuming no major mistakes or criminal acts have happened. How long to stack those boxes? How long to make those cakes?

Most creative occupations don't have this. A few do; if you're a professional writer taking up front distributions, you'll be asked for dates, by example. But all told, it is relatively rare for creatives to be deadlined, because it's so unreliable.

One major issue is that creative work is rarely collaborative-cumulative. There aren't many jobs in which one creative is stuck because the other creative isn't finished yet.

Collaborative jobs tend to want scheduling so that one staff member doesn't become overcommitted. It's relatively easy to understand at a lumber mill: you need one person doing center-sawing for every person doing bark shaving. If you have eight of the second job but only one of the first job, seven of your people will be generally idle.

Software has a core problem, in that pieces which are dependent on one another are difficult to schedule. There are very few processes that work this way, so society doesn't actually have very good norms for this.

At this point, opinions are going to split on how to interpret things.

Many people will say "well, someone has to schedule." This isn't actually the case, of course; I've been at software shops where the software is done when it's done, and everyone can basically screw off if they want to know more about what's ready next month. This is more common in SAAS shops than in traditional shops; SAAS shops don't usually advertise on next month's features, and don't have to encourage users to upgrade.

Many people, charging forwards assuming that the scheduling must occur, will say "well what, do you want it to be the managers?" And yes, I actually do. They do in every other industry. "But they're not doing the work!" I don't care. They don't do the work in any other industry. "But they don't know how long it's going to take!" Again, I do not care, because I don't either. To me, the entire reason this seems like a problem is punition.

"Is that even a word?" Yes; to leverage it is to be punitive. To punish.

Imagine, if you will, with me, a world where a manager is hired to be able to manage, rather than promoted out of being a good developer.

Imagine a world in which literally nobody is held to the estimates, so we don't have to pad them and juggle them and make them nonsensical to protect ourselves. They can just be best guesses.

Imagine a world where the estimates are made by a third party who's been watching you for a year, who is trained in statistical estimation techniques and is data driven.

Imagine a world where you didn't make the estimate, and if the estimate is wrong, nobody gives a shit. It was just an estimate.

You know what you just imagined? Every other job on Earth.

Go talk to a construction worker. Oh, the estimate was short a month. Are you doing overnights to get it in on time? Lol, no. What professional would do that?

Lawyers? Not unless a life was on the line.

Doctors? Not unless a life was on the line.

Actors? Not unless ten million dollars for them personally was on the line.

Who would do that for their regular salary?

Well, computer programmers. But that's because we've all been taught these nonsense rules by "senior" 19 year olds at their first job, and somehow, we've fallen for them.

These things are happening for one very simple reason. Just one, and only one.

Because you aren't saying "no."

It's very simple, it turns out. Go into that room with your head held high, and say "you know, the science says that making these estimates slows everything down, makes everyone unhappy, makes people make bad decisions on things that are little better than guesses, and drives your best programmers off of the team. FAANG hasn't done this for almost ten years. Smart bosses are moving past this."

Then just stop there, and wait for your boss to want to look smart.

Get one of your coworkers to do the same thing, three months from now.

It'll take six months.

Your boss doesn't want to do this either; they just think they have to, for the same reason that you do.

Lead them to Elysium. Freedom is possible.

There's a thing in psychology called the monkeys and the ladder. This is a real thing, and it's just one inch short of abusive to the animals, so it's hilarious.

Get five monkeys in a room. Put a ladder in the room. Put some awesome food at the top of the ladder.

Every time a monkey goes near the ladder, start yelling, from a safe place behind a fence or something (because monkeys will mess you up.)

If a monkey touches the ladder, hose it and all the other monkeys down. They don't like that, in the way that cats don't like that, and they will bail.

After a half hour of this, they will figure it out, and all the other monkeys will stop any given monkey from using the ladder.

Now, replace a monkey with a new monkey. They won't let the new monkey use the ladder. The new monkey will never see the hose.

On each following day, replace a monkey. A second. A third. A fourth, and now the fifth.

Now you've got a whole new team of monkeys. None of them have ever seen the hose. All the same, they won't let each other use the ladder.

This is your management team and its estimates.

Go use the ladder. They need to see that it's safe before their tiny simian brain can adjust.

Tell your management team that if they unburden one team from estimation for twelve months, that team will radically speed up, become happier, feel less scared, and produce more side projects.

Tell a manager who isn't the oldest, so they can ladder climb, but isn't the youngest, so they have some weight to swing around.

You'll see.

Why do we do this?

Because ladder tradition is hard to shake. Because few of us have the courage to just try going for the goddamned banana. Because we think in terms of what the previous monkeys insist on.

There is no good reason for programmers to estimate, and all the research science says that it's a terribly toxic and counterproductive pracice.

Beware any Lumbergh who patiently explains.

Answer 12

There are common bad practices around unscrupulous managers trying to extract deadlines from often very inexperienced developers, that much is certainly true.

But let's approach the question as if everyone were only acting in good faith.

For the manager, they always have a legitimate interest in some kind of estimate, because it will influence decisions about the allocation of resources (and the justifications for it).

I think the problem is not always that experienced developers cannot estimate at all. Rather, experienced developers know two things in relation to estimation.

One is that estimates typically consist of relatively large ranges. And two is that there is always some residual uncertainty, the cause or likelihood not foreseeable in advance, but past experience showing that sudden hard stops can be encountered.

Many managers, not being drawn from an appropriate background, often cannot understand why ranges are so large, or why there can be unforeseeable failures.

And the topic is sufficiently complicated that it's difficult for professionals to discuss amongst themselves, let alone educate a non-technical manager.

A task that may take an hour or a day is not typically problematic for a business, but one that may take a month or a year, and may fail after all, is seriously high stakes for anyone responsible for funds.

That's essentially why software developers are not only expected to estimate, but also why even managers acting in good faith are loath to accept the estimates made by developers. Because the estimates are far too wide, and the developer cannot explain why it must be so wide.

Software development is unlike most other kinds of workplace activity, in that you can ask a relatively experienced person to start a task, yet he doesn't know whether it will take two weeks or six months, or whether he will even succeed at all.

Another aspect of this problem is that, whilst most software development tasks are not completely unfamiliar to the software development profession as a whole, businessess typically commission their developers with such a wide variety of tasks that they rarely have all relevant knowledge already amongst their own staff. Most developers are solution-finders and solution-makers, not solution-knowers.

Businesses could often reduce uncertainty in estimates by hiring more expertise, but the difficulty and timeframe of recruitment and the cost of the labour would completely dwarf the estimation problem they were trying to solve in the first place.