What's the best way of writing user stories about "improve this metric" if I'm not sure how much improvement is possible?

Question

Say you have a piece of software that you know can be improved on various quantifiable metrics (time to boot, CPU and memory usage, latency for various actions, accuracy of recognition tasks, etc.), but you don't know by how much - how would you write User Stories?

Is it better to guesstimate an improvement? (ideally with the team) To just write "improve such-and-such metric", without specifying by how much?

(Our organisation is new to Agile and we're trying out Scrum, and I've found we spend a bit too much time discussing these details and the meta topic of how to formulate them, so I'd like to know how others would cover this; we don't have specific formal requirements from our client on those, but know (from UX testing etc.) that those improvements are needed for a better user experience)

Answer 1

Performance improvements are often an exercise in cost-benefit trade-off. The executive who "just wants it to be faster" might find adding one server to boost speed by 50% an acceptable cost, but not two more servers for the next 20% boost. Also, there are usually many possible technical approaches to boost performance -- code refactoring, infrastructure changes, switching out components. So some degree of research and exploration is often required to discover the optimal solution for your current technical and business situation.

I would suggest you start with a time-boxed spike to brainstorm, research and possibly experiment with some possible approaches. (Sometimes this can even reveal some no-brainer quick fixes.) Then report back to stakeholders about the options, each with rough estimates of cost (in time or money) and expected benefit. Then a decision can be made on which to implement. At that point, the stories can essentially be in the form "do option B." Make sure to benchmark the metric before and after to determine if the expected costs and benefits were on the mark. Then a decision can be made about whether to proceed with additional steps. 

Answer 2

TL;DR

You can't invent meaningful targets out of whole cloth. Pragmatically speaking, both business analysis and user experience requirements all require forming a hypothesis, testing the hypothesis, and then drawing an actionable conclusion (e.g. your specifications) from the results.

Your team (and particularly your Product Owner) don't appear to have the right processes in place to gather meaningful specifications about non-functional requirements related to the user experience.

Gathering Requirements is a Real Job

My question is precisely about how to make good stories. Say you're making a game, and starting a level takes two minutes, which frustrates playtesters. The user need is concrete ("goddamit I want it to load faster!") but vague.

Good requirements gathering often entails experience in performing multiple drill-downs until you get to actionable specifications. "The load time is too slow" is not an actionable statement. However, a management target based on industry research (e.g. "load times must be less than 8.3 seconds to maintain engagement") or actual experimental data can certainly be turned into user stories.

A Product Owner should be working with Business Analysts (BAs) and User Experience (UX) designers to construct tests. For example, the UX team might develop a minimum viable product with load times of 5 seconds, 10 seconds, and 20 seconds. A good UX team would then gather data in the form of opinions or testable human behaviors to determine how "sticky" the product is at different load times to see what level of load time leads to actual disengagement or lack of interest in the product.

User Stories Are Not a Substitute for Understanding the Knowledge Domain

User stories are conversational placeholders between the development team and the Product Owner or end users. They are not specification documents, nor are they a substitute for essential business skills. A great cross-functional team may include UX expertise within the team, but most often requirements-management of this kind is really external to the team, and proxied by the Product Owner.

In other words, your problem isn't that you are having trouble crafting the user stories. The real problem is that your team (and particularly your Product Owner) doesn't have the right processes in place to gather meaningful specifications about non-functional requirements related to the user experience. Whether internal or external to the team, the organization needs to ensure that those processes exist and are properly hooked into the product development process.

Answer 3

Often these types of stories come from high level stakeholders - in our case it was the CEO complaining the site is too slow. He was asking for sub-second performance, which was not realistic due to the data processing involved.

We used vague stories such as 'Improve performance of X', which gave us enough of a goal to identify some different technical stories that were all likely to improve performance. We then prioritised these based on effort and likely impact. We then started at the top, until we got to a point that our product owner was satisfied we had made enough of a positive impact, and the remaining effort outweighed the benefits.

I don't like placing a concrete target on these sorts of things, as you may later find out you can get 90% of the way with 10% of the effort, and that is good enough. If the work you have done hasn't made enough of a difference it will definitely be raised as an issue again and you have confirmation you need to do further work.