How to address external problems within our issue intake process?

Question

We're trying to put together an efficient intake process for bugs/issues that are encountered through customer service, etc. We haven't had this before.

Many of these issues are external analogue systems & human factors that aren't part of our company – they aren't things we'd expect the user to know or understand. They just know they're using our product and things didn't go as expected. I know we'd have to do discovery and determine whether it's our issue or external, or if we can mitigate the problem – once someone has reported an issue through our intake form.

How should external issues like these be addressed?

Answer 1

The Underlying Issues

[T]hey aren't things we'd expect the user to know or understand. They just know they're using our product and things didn't go as expected.

This is likely a documentation or user interface problem. When you sell a car, the manual tells you how to use or operate the car, but it doesn't teach you how to drive. It also won't tell you how to drive to work unless you've built in a GPS system, in which case addressing the settings of the GPS are your problem, and any faults in that subsystem system are yours, too.

The car analogy is a leaky abstraction, but the point is that you need to communicate more clearly with your customers about what the boundaries of your system are, and what problems are "yours," what problems are "theirs," and what problems are "someone else's." A reasonably good analogue of this is the AWS Shared Responsibility Model that highlights what Amazon is responsible for and what they are not.

In addition, you need to design your user interface better in order to more clearly delineate what your product is supposed to do, and what the customer can do to troubleshoot or categorize issues themselves. Any sentence that includes statements like:

[These] aren't things we'd expect the user to know or understand.

are inherently problematic. It either means the customer doesn't understand what was purchased, the documentation is unclear, or the user interface doesn't expose necessary troubleshooting or configuration options. Fix those things!

Focus on Consistent Accuracy, Not "Efficiency"

Unless you address the root cause, which is apparently that your customers don't understand your system or its limits, or the complexities of how it interacts with third party systems—all of which are problems your company should own in terms of design, documentation, or a more upfront sales process—you will forever be saddled with "not our problem" bugs. The air quotes are because the product is part of the problem here, but the company is trying to say it's due to a third-party dependency that customers don't understand.

If you choose to take that approach, most companies define a process that identifies classes of problems that are "as designed," and you will commonly find bug tracking systems with labels that effectively say that this is the way the product is supposed to work, or that they are problems that will not be fixed.

If you search Stack Exchange or Github, you will find many sites and projects with labels such as:

• "by-design"
• "wontfix"
• "not-reproducible"
• "external"
• "upstream"
• "unsupported"

While these are perhaps the most common ways to tag such problems, you can label issues any way you like. I'd also recommend adding labels for specific process/product components that you can address as an organization such as:

• "documentation gap"
• "self-service gap"
• "troubleshooting gap"
• "user interface issue"
• "product communications gap"
• "incompatible or undocumented use case"

or whatever else you feel is really at the root of calling a feature (or even a misfeature) a bug. If you can't quantify the work involved, nobody outside your team will care.

While politically infeasible, adding a "lying weasel making false promises to meet sales quotas" tag is not always inaccurate, either. This happens, but even then the reported problem can still be placed into one of the other buckets without creating political friction, so keep that in mind.

Where's the "Efficiency?"

The process above will provide feedback to the organization and the product development teams about how to improve the product so that you get fewer issues reported that you can't do anything about. In the long run, this will result in fewer cycles spent categorizing things that can't or won't be fixed.

The other improvement will come from having a documented triage process or filter that ensures that less time is spent reinventing the wheel every time you analyze an issue. This can be facilitated by collecting common externalities into the issue reporting process, or by documenting that problems of a certain type are automatically tagged as "wontfix" or similar.

Here are a couple of examples.

Efficiencies from Intake Filtering

Thank you for reporting an issue with Confusing Products, Inc. Before submitting the issue, please check all appropriate boxes:

1. [✓] I am using the product with Incompatible System A.
2. [✓] I have examined /var/log/confusing_product.log and no errors were found. Log is attached.
3. [   ] We are not using the product from Confusing Products, Inc. as a food processor.
4. [✓] We are not using the expensive "user incontinence" add-on from Confusing Products, Inc. as swimwear.
5. [✓] We did not use the exposed 20,000 volt electrical wires from our Confusing Products, Inc. system to perform home electrolysis.

Doing this will help you quickly determine that the client was using the product as a food processor instead of as an ECG machine, which will certainly improve the efficiency of your categorization process. It will also help customers determine what troubleshooting steps they could take themselves, such as checking a list of incompatible systems, unsupported use cases, or other known wontfix activities.

Efficiencies from Categorization and Labeling

The stuff that does get through such a filter will narrow down the problem, and help you use your documented process. Some useful outcomes may include:

1. Labeling issues more promptly so you can give appropriate responses or feedback more quickly.
2. Redirecting customers to existing documentation or knowledge base articles.
3. Fixing identified issues that can be fixed.
4. Improving the self-help tools available to customers.
5. Updating the FAQ, documentation, sales materials, or other informational material to avoid having to continually re-solve the same issue.

In other words, any improvement in efficiency will come from handling fewer avoidable reported issues, and an improved level of issue screening with a clear separation of responsibilities (e.g. yours vs. not-yours) in the user interface, the documentation, the bug reporting process, and the way the company sets customers' expections.

Limitations of These Approaches

Neither of these approaches will solve the problem if the company refuses to improve the product, self-service, or customer communications, but if that's the real problem then no amount of triage will make a defective process more efficient. The best you can do is to provide feedback to the company and its users about what is or isn't a bug, and to suggest ways the business can improve before customers vote with their wallets.

Your goal is to filter out non-bugs (however you choose to determine that) as quickly and repeatably as possible. Your secondary goal is to improve the accuracy of your issue handling, so that appropriate metrics can be designed around the issue-reporting and bug-handling processes. What middle management or the executive leadership team chooses to do with that information is, quite honestly, above the pay grade of most project management professionals, and is ultimately the responsibility of the C-Suite and the Board of Directors that empower them. Do your due diligence; the rest is up to the people who run the company rather than the development team, the support team, or the help desk.