How do you measure progress for design works?

Question

We are a software agency that adapts product approach because some of our clients need a discovery process before we actually dive into coding. This is a typical situation when a client comes to us with a pure idea for a software product and expects us to provide an exact price and deadline for delivery.

We take one step back and provide a service where we define the product first. This however includes some design works, where we create a visual prototype.

As a product trio, we start with user-story mapping, core user flow, and outlining the necessary screens. The output of this stage is a list of screens (incl. the user stories), which serves the designer as the scope of work (a design backlog), which you can imagine somehow like this:

• Wireframe of the Homepage (+ some user stories)
• High-fidelity of the Homepage
• High-fidelity of "My Account" (+ some user stories)
• ...

After this, the designer estimates the effort needed to deliver these screens, so we can forecast the deadline if no change requests will be needed.

Now here comes the struggle:

The velocity of burning down the design backlog is not exactly linear - it increases as more components are recycled and the design system is clearer. Also, the design works don't actually progress sequentially - sometimes the designer has to go back and forth between screens or go parallel.

When we ask the designer about the progress, we usually hear something along the lines of "I think we are on track and the deadlines are achievable." Even when 80 % of the time is gone and only a small fraction of screens is completely finished. And then, 3 days before the deadline: "I think I would need some extra time." The budget was being burnt accordingly, but when the deadline is about to come, we are not finished.

I tried another approach when I divide the number of screens by the number of working days and get the "number of screens per day". Then I set up some milestones, what should be done. I feel it's kinda toxic and it also doesn't work, because the designer typically claims that he has some work in progress and the next milestone will be fine.

When the designer is asked, what should we use to identify that we are off track, we usually get a response like "I don't know, it's very hard to estimate the design work as it's very creative, sometimes it's harder than else, the first screens are harder than the late ones..." - no solution.

TL;DR: What system of tracking progress is healthy for a graphic designer and also sustainable for the company? Is our situation typical in other organizations or do you see the potential problem?

Answer 1

TL;DR

Unless you're using standardized templates with minor variations, your designs are most likely net-new. That means means that you don't really have a good handle on "yesterday's weather" to make accurate forecasts. You're essentially trying to build something that has an unmeasured cone of uncertainty without devoting time or effort to the actual estimation process. This is a process that is designed to fail.

Analysis and Recommendations

Your current process exhibits the following challenges:

1. Follows the "big, upfront design" (BUFD) pattern with an expectation of estimation accuracy.
2. Frequent task-switching for a single developer, which creates significant overhead.
3. An all-or-nothing delivery strategy.
4. Tightly-coupled deliverables that must be delivered in their entirely and in tandem.
5. A specification-based plan, rather than an emergent design with flexible scope around implementation details.

These are common challenges. In agile frameworks, "plan the work then work the plan" is an anti-pattern. Instead, a key goal for agility is to leave most non-essential design decisions to the last responsible moment. In addition, scope should be the flexible constraint; delivering some essential value within a given time box ought to take precedence so that the client can say This is good enough! or This is not yet fit for purpose.

So, here are some key recommendations that will help.

1. If possible, define a baseline product or set of products that require refinement and tuning rather than entirely net-new design work.

2. Use the INVEST mnemonic to deliver smaller, vertical slices of partial functionality for rapid feedback while the work is still in progress.

   NB: This not only creates opportunities for evaluating product fit, but it also creates a chance to reduce unnecessary scope or declare the existing set of deliverables as "good enough."

3. Value can and should be delivered in smaller chunks at a predictable cadence.

   As you've seen, having "80% of everything 60% done" isn't a very useful metric. Queue and batch theory basically rely on smaller chunks of a reasonably similar size. This generally leads to a more predictable cycle time where things are either "done" or "not done."

   The upside to this approach is that you have a new and measurable way of determining actual progress rather than guesstimated percentages. The downside—and there is one—is that it requires rethinking your workflow, estimation process, and (in particular) performing sufficient decomposition of the work so that it meets INVEST criteria. Unless you are able to break work down significantly further than the all-or-nothing delivery process you currently have—this seems based on strong inter-dependencies between the deliverables, which usually means they're currently too tightly-coupled to deliver in independent chunks—then you will not be able to deliver increments of value. You are then stuck delivering either all the value, or none of the value.

   This looks like an estimation or effort problem, but it's really more of a process problem. You need to fix the process, and the way you're planning and delivering the work is a big part of that!

Pragmatic Client-Facing Recommendations

I'd also take this a step further, and stop offering fixed-price, fixed-scope work. The Manifesto for Agile Software Development values:

Customer collaboration over contract negotiation[.]

That means that rather than treating everything as a fire-and-forget customer request, you should actively be working with customers on a routine (if not ongoing) basis. That not only enables you to refine the deliverables so that they are a better fit, but it also enables more transparency into why certain things are difficult or time-consuming. This allows for closer collaboration between the designers and the customers to find solutions that are faster and easier to implement, and not simply done a certain way because it was something discussed during project initiation.

Answer 2

The problem with software design is that it involves the production of something fundamentally new in character.

The exact shape of the result is not known until the design work is complete - the work of software design is concerned with specifying that shape (in enormous detail) in the first place, so that the resulting design is suitable for its purpose.

It could be thought of like old-style oil prospecting. You can't measure how close you are to the oil, because the work of prospecting is concerned with answering the prior question of where the oil is located. An element of speculation and uncertainty is inherent.

There are sciences, and inferences from past experience, which are capable of improving predictability and success, but these are expensive facilities only available to large scale operations, and even then do not guarantee success of every adventure.

Obviously software design is not oil. The analogy concerns the fact that the work in each case is about searching and finding out something you don't already know.

You can't measure things you don't know, because measurement is fundamentally about comparison of known things. Comparison to a measurement standard. There is no comparator for a new design, because it is usually a one-off bespoke production, and therefore there is no past experience of its production or extant standard against which to measure the progress of its production.

Each new software design poses different puzzles or problems to be solved. Typically the detailed puzzles or problems only become apparent in the course of doing the work and becoming deeply familiar with the requirements and trialling solutions, and they require a variable and unpredictable amount of work to solve.

What are first seem like subtle or minor changes proposed to an existing design, can sometimes entail widespread changes as a consequence, and therefore large amounts of design work and problem-solving.

Sometimes irreconcilable contradictions are discovered, which either scuttle the project or require some other part of the system to budge. The "system" being potentially anything that can be the target of human attention.

Summary

Think of software design work as typically making the unknown into the known, and of measurement as comparing the already-known to the already-known.