How do we measure design quality? This question has been asked at every company where I’ve led design, and it comes not only from designers but from stakeholders and executives as well. Solutions range from “we’ll know it when we see it”, to principle-based evaluation, to detailed and rigorous product and design reviews. Other suggested solutions have included surveys like NPS (Net Promoter Score), CSAT (Customer Satisfaction Score), or custom user surveys like “How delightful do you find our product?”
Each of these solutions is problematic. Internally-led evaluations may be biased by internal motivations or blindspots. Surveys may not be able to isolate design quality, or they might add an unnecessary burden to the user.
A few years back, I was leading a design and research team at a company that was newly obsessed with John Doerr’s philosophy and application of OKRs. We found ourselves measuring everything, including design quality.
The evaluation mechanism we landed on was an expanded set of product and usability heuristics based on Jakob Nielsen’s 10 Usability Heuristics for User Interface Design. We came to the conclusion that while there may be bias in a design team evaluating their own work, it’s a no-brainer that designers should know what “good” is, as critique is an essential part of our practice.
I worked with a few teammates to develop and iterate on a set of heuristics mapped to our general breakdown of product design: product thinking, UX design, and UI design. An additive benefit was that we looked at designer qualifications and skills in similar groupings, so as we were evaluating quality, we could easily have input to the question “where has the designer exhibited strengths or areas of opportunity in this project?”
To evaluate, a small group of us sat down after every product launch and did a lightweight review of the work that was shipped. We answered a simple yes or no to each question and implemented a formula to give a quality percentage score (100 for yes, 0 for no).
While of course, this wasn’t a a perfectly precise conclusion, we logged resonant variants from project to project and found it useful to have the language to give feedback to designers as well as share a state of quality with non-designers.
Here are the heuristics.
Clear task at hand
Straightforward and ease of use
Recognition over recall
As a note, these were customized to our beliefs on design quality, and might vary based on context. For example, design principles were the principles that we established as a design team, and we focused on how people learn as we were working with students.
While these worked for us, we acknowledged that it wouldn’t scale much past our small team; for example, a team with hundreds of designers shipping daily wouldn’t likely have the bandwidth for manual reviews. However, as designers learn to moderate their own craft in different areas, we believed that these would still reinforce quality, and that implementing these formally or informally would still lead to quality output and outcomes—feeding into measurements of product and feature success—for both the company and our users.
Thanks to jennie § yip\ and Casey Callow for the partnership in designing these!