How to measure the impact of design
Tangible ways to turn product interaction data into measurable systems for user experience
This blog post is in collaboration with UXDX, and can also be viewed on the UXDX blog site.
When it comes to product data, teams track quite a bit related to the health of the product like down times, number of ticket, number of customer calls etc, but what about the user's experience in interacting with the product? While experience measures like UMUX, diary studies and other quant/qual methods exist, they are highly manual and usually require a third party tool or commissioned report.
What is the problem?
Organizations often chase metrics that are irrelevant to the user experience, or have a "catch all" proxy metric (like NPS) that stops at a superficial level. Companies are pressed to measure the user experience, but often misunderstand the complexity and richness available in true data, and thus latch onto an easily quantifiable number (i.e. NPS). Even if a company does track user experience metrics, the approach is usually scattered and there is usually no systematic place to track the data.
Device telemetry and instrumentation rarely include design-data standards as part of a requirement during set up, meaning clearly established, user-centric metrics are missing from the software development process.
The consequences reach further than poor UI:
Gathering design impacts is often reactionary, with designers scrambling to justify decisions or use indirect measurements as a proxy for their team's work instead of being planned ahead and instrumented in code, like other product metrics.
Even if a product team wants to measure the impact of design, the process is manual and expensive, so rarely done.
Products face reactive decision making.
Because decisions are reactionary, there will never seem to be enough researchers or designers, and budgets will always fall short of what is truly required to measure usability. In fact, any research budget actually approved is saved for "truly important" insights, and product teams are left in the dark between these large-studies.
Finally, when measuring is not well integrated into the tools and practices and systems, it creates burnout. Researchers and designers will often find themselves reproving the same point, or the effort of measuring usability might get wasted and gather dust in some sort of drive or a slide deck presentation. This leads to attrition, this leads to apathy, and it leads to burnout.
Measuring Data Challenges
Measuring quality is hard: it requires new skill sets as well as the creation of new roles
Technology is going to become more complex and integrated. As designers, we'll need to question how our design systems and tools feed into this
Design already digitizes faster than we can measure: we need to understand how to automate the measurement of design quality
Most frameworks are missing design centric data requirements. And so they're not well integrated into development practices. So, even how usability factors relate to each other is often left to subjective means of understanding or ad hoc guesses.
Measuring Design Quality
Measuring quality is nuanced. There's a difference between subjective and objective measures of quality. However, both can be measured with the same category lenses using the three steps of defining factors, criteria and finally metrics.
1: Determine Your Design Factors
For example, you can determine the design factors that make up quality. We're all familiar with the big three of efficiency, effectiveness and satisfaction. But over the years, other factors have become dominant in the field, such as trustworthiness and safety, both data safety and physical safety.
2: Outline the Usability Criteria within the Factor
Next, each factor can be broken down into criteria. What is efficiency composed of? When things are efficient, they're understandable, they make really good use of your time, their operations are perform easily and their resources are optimized.
Let's take understandability as an example. It refers to how capable a product or a software program is in giving clear instructions and support. Breaking this down into criteria, we can ask some of the following questions:
Do our components have the right micro-copy?
Can we measure how many times someone has clicked on a help link?
These immediately draw distinctions between a qualitative state of something being understandable where, you know, the user is saying, yes, I understand what you need me to do. And it actually becomes a back end indicator that your researchers can use before there is actually a problem. It is the red light signal that tells you, look here, there might be an issue.
So as you define the design factor and the subset of criteria, you start to create a matrix. You can see at a glance the product characteristics that are important to how you evaluate the interface. You can identify overlapping areas. Now, these are just some of the outlines. Breaking down all 10 categories creates about 100 plus combinations, but you can see where there is overlap. What could we measure once that actually encompasses more than one design factor?
3: Match Your Metrics to Your Criteria
Criteria like time and productivity or learning, they are made up of something. They don't just exist in a qualitative state. What becomes an abstract concept can then become a logical sequence of operations that is used to quantify a quality attribute. You have to define the numeric value. That's the next step. For example, one of the functions of time is Minimal action. How do you define minimal action? Criteria could include task time, where the metric is the average time for users to move through a product funnel.
This moves you from qualitative to quantitative research because once you've defined your design factors and the subset of criteria, you start to create a matrix and identify overlapping areas, and then grade them by importance.
While this may seem daunting, keep in mind that it's really no different than how design leaders and design teams have broken down the compilation of design systems into digestible qualities for our products teams. We do it all the time. But we use words like atoms and molecules, instead of metrics, criteria and factors. But it's the same in a design system which we're familiar with.
4: Gather Data from Your Product Team
This is where the fun part starts, if you think data is fun. Most of this data will exist in some shape or form, it just won't exist in the same place. It might be in a business format or lost maybe within a monthly operational review. This is where your quantitative researchers, your data analysts, or your statistician become your best friends. They swim and live and breathe in this data every day. So partner with them.
If you, as the designer, know the insights that tell you the experience quality, and you can articulate the specific dimensions of information that you need to your product person, you can find the information.
So, what started out as sort of like a mad game of find a needle in a burning haystack that's somehow also in space, turns into a more logical approach of a gap analysis.
The benefits of measuring data quality
It can reduce the cost of usability testing. Measuring design's impact accurately reduces the cost of usability testing, if we provide a consistent basis for understanding and comparing different usability metrics at scale. For example, I insisted that the product team develop and track time on task, completion rate and throughput. By doing this, we reduced the lead researcher's time to provide a benchmark report down from two weeks to only half a day.
It compliments expert-based evaluations of usability with fast data. Moment in time research that is quantitative is expensive, and researchers struggle to find large enough sample sizes to validate evaluative findings. Calculable metrics are the result of mathematical algorithms or heuristics based on raw observational or countable data; this doesn’t happen naturally in product development. It has to be built into the development cycle. But doing so compliments evaluative experts.
It lays the groundwork on clearer communication of usability measures between your design experts and the developers. In one instance, automating the measurement of time on task meant the developers could monitor improvement without having to rely on a researcher to present a report.
It democratizes usability measurement practice (and automation). Democratizing usability measurement also creates a foundation of consistent measures that are part of telemetry and instrumentation. Engineering, design, research, and product can all gather and rally around a common understanding.
Applying Data Challenges
All data needs to be taken with a grain of salt because context is important. How humans are using this really matters because usability without context of use is meaningless and design without the context of humanity is useless.
As humans, we want to know things. We are drawn towards numbers we want to quantify our experience we want to know.
Measures should reveal what we really need to know about design. We don’t just need to know NPS, we need to know NPS and accessibility. We don’t just need to know sales, we need to know sales and productivity. This reflects an internal quality that ultimately reflects your design philosophy and strategy.
We need to measure what is worth understanding. Worth is a balance of value over cost, yet too often we focus on the cost and fail to quantify the value. And what are we doing as designers, and as design leaders, if not building value for our users? What are we doing if not creating value for our future?
Comments