Usability Testing: User Research at the Center of Duo’s Design Process
From the early days at Duo, we were interested in developing great products and providing a great user experience. Usable systems are not only more satisfying, but they're also actually more secure because people are more likely to use them.
With this in mind, a year ago I was tapped to create an in-house rapid-cycle research program based on frequent usability testing. Starting when we were relatively small—only about 150 employees—we began to have five actual users try our products in development on a biweekly basis aligning with our agile design cycle. Since then, more than 100 testers from a variety of industries and geographic locations have participated in our qualitative research program.
It's rare that a company would bring user research competency in-house and decide to invest in it at this very early stage. It's an acknowledgement of how dedicated we are to our mission of making security easy, that we want to invest in user research in a meaningful way, long-term from the very beginning. - Sally Carson, who leads product design and user research at Duo.
Analysis of usability tests at Duo involves designers, developers and researchers. After each session, the team typically notes on a whiteboard what was learned from each tester’s progress on tasks.
Here's how a testing session works. With a moderator leading and other Duo team members observing online, each test subject tries to accomplish activities on prototypes for about a half hour. Customers who have experience administering Duo try out new admin tools. End users typically kick the tires on user interfaces. Recently, testers have explored information architectures, modified policy options in Duo Access, tried out our German and French interfaces, and updated device settings in the Authentication Prompt.
Duo makes usability testing engaging, and their tests and examples show how they are trying to bring new features in the best way possible. Duo was very supportive and encouraging while I was engaged in their tests. - Kevin Hayes, Director of Information Security at Wayne State University.
For some studies, team members provide observations on paper notes to be clustered for analysis.
The research pays special attention to the actions the testers take and whether they succeed at the activities they are asked to do. However, testers are encouraged to think out loud as they go, which gives a sense of what is confusing, what testers expected to happen, and why they chose the paths they did—in effect, providing an explanation of their behaviors.
Testers reveal prototype elements that are highly usable—typically learnable, efficient and effective—as well as those that aren't. In particular, the Duo team looks for problems that multiple testers encountered or more severe issues that affected a smaller group of users. Our product designers incorporate the findings into their next designs. Additionally, the research team shares lessons widely by showing video clips at regular engineering full-team meetings.
Usability testing with Duo was a great experience; it was easy to talk with the administrators of the testing, and I felt especially helpful being able to share my perspective as both a user and an administrator. I have enjoyed using the product, and being able to contribute to the future developments was fantastic! - Kendra Strode, a computing support specialist.
Duo engineers, designers and product managers attend the testing sessions for products they are developing, allowing them to experience users' frustrations firsthand. They also share their observations and analyze the sessions. The process is lightweight and leads to teams sharing different perspectives. It also exposes each new observer to user research best practices, analysis and our iterative design process.
Duo also experiments with various analytical methods, including lengthy end-of-study discussions captured in a detailed report by the moderator, whole-team affinity mapping and whiteboard discussions of findings after each test (a method suggested to us by Michael Margolis, UX Research Partner at Google Ventures).
Customer feedback and usability are important to Duo, and they listen to their external testers as they improve their products. I applaud Duo for taking the time and effort to include their customer base in usability testing of improvements to their product. - Marc DeBonis, an IT director who has participated as a tester.
How Can I Participate in Usability Testing?
Email uxtesting@duo.com and tell us your job and which Duo products you use (such as Admin Panel, Duo Mobile, Authentication Prompt or something else). If you're not currently a Duo user, that's OK; just let us know in your email. We'll reply to schedule you into a study.
Usability Testing FAQ
What do I need to be a tester?
Just a PC or Mac running a Chrome browser, and a phone.
What preparation do I need to do for a test?
None; simply come to the online meeting at the scheduled time, ready to share your thoughts!
Why do we test on only five users?
Our approach is qualitative and isn't meant to be statistically significant. A few users provide ample indication of potential problems our designers can tackle.
Why are the tests done remotely? How is that handled?
Most testing happens during daytime work hours, using common online meeting software. During sessions, participants are usually in the environments where they interact with real Duo products day-to-day, so their conditions echo reality better than a usability lab setting would. Testers sometimes need to pause to answer phone calls, chats or in-person questions from co-workers; we welcome these interruptions, as they help us understand the conditions users face, and we empathize with them. Remote testing also allows us to test with people from several time zones in the same day.
Are all of the tests 25 minutes?
Most are, but when testers ask to go longer than the scheduled time, we're glad to oblige if doing so doesn't conflict with other scheduled tests.
Some companies offer an incentive to testers. Does Duo?
Yes. Each tester receives a $25 Amazon eGift card as a token of our appreciation. Participants outside the U.S. are eligible for cards from Amazon's international sites, such as Amazon.co.uk.
Does Duo test on existing products?
You bet. Testing on current features helps us preserve interaction patterns that work well, and that can serve as proven starting points for new features.
What doesn't this program do?
Testing only a few users doesn't definitively inform us about what actions users in general would prefer to take. It also doesn't provide us with clear guidance about visual design, as that feedback often is highly subjective. Behavioral testing also doesn't tell us much about how valuable or satisfying a feature might be, even if testers say they like a particular design. To research these areas, Duo uses a variety of other UX methods.
Are testers' comments shared publicly?
The sessions are used only internally to improve our products. Interestingly, many testers ask to supply additional ideas to our Product or Marketing teams—in essence carrying the spirit of product research beyond the confines of usability testing. Our testers sign an NDA/Consent to Record form, as we strive to be transparent about what we record and the nature of the insights discussed.