The Right Test at the Right Time

A/B testing is often spoken of as though in opposition to usability testing. In reality, the two types of tests answer different questions and serve different purposes. Both have a place in a project; in fact, they often complement one another.

It is easy to find arguments for one method over the other. Some swear by usability testing alone because it gives insight into the client’s needs. Others claim that only A/B testing is useful because it offers concrete numbers and can be done on the cheap. Both sides believe that one test is superior to the other. Even UX guru Jakob Nielsen has written on the subject – as he referred to it –“putting A/B testing in its place.”

But Nielsen’s article hits upon something crucial that many other UX practitioners overlook: usability testing and A/B testing each have a place; furthermore, both can be used in one project. Most projects would benefit from doing both types of test, but even if only one can be done, it is necessary to understand what each testing option provides, and how (and where) it fits into a project.

Strictly Quantitative: How Much or How Many

A/B testing involves taking two designs and setting them up for comparison. This provides some obvious benefits:

• Flexibility: A/B testing can be as granular as changing a single button icon, or as overarching as creating two completely different sign up pages.
• Live testing: A/B tests can direct half of all users to each of two variations, and measure the resulting purchase, conversion, and bounce rates.
• Quantitative Metrics: Data such as growth or loss of sales percentages, subscribe rates, and bounce rates are easily measured.
• Low Maintenance: A/B testing can be very inexpensive and require minimal effort in the long term. Once set up, the test can be left alone for weeks or months until results become apparent.

There are also some key ways in which A/B testing differs from usability testing:

• A/B testing answers the questions of “how much” or “how many” and even “which one is better” but it does not answer qualitative questions, such as “why” or “how.”
• A/B testing is cheap once it’s set up (no maintenance necessary), but requires more up-front prep, such as two complete and live variations of the same page.

So when should you A/B test? When you want to compare two or more things, but are not sure which will be more effective. A/B testing is in its wheelhouse when two designs or copy exist, both with obvious benefits, but with different focuses. For example: one design presents a search bar front and center to help users find content on the site, where the other encourages the user to subscribe to the site early on, making the “sign up” button the most prominent aspect of the page. The question: will users pay for content because they can easily find everything they need on the site (thanks to the search bar), or are they more likely to sign up when asked straight off the bat?

To Gain Context – the Why and How

Unlike A/B testing, usability testing is most often labeled as “qualitative,” because usability cannot be measured purely by numbers. Questions such as “how can that be accomplished?” “why is this frustrating?” and “where is the sticking point?” reveal the user’s thought process – something that cannot be reduced to statistics. However, relegating usability testing to only qualitative is a disservice to its value. What are some of the benefits specific to usability testing?

• Immediate Results: Usability testing can generate results with as little as a partially-complete mockup, prototype, or wireframe.
• Quantitative Metrics: Usability testing measures metrics such as task completion and duration, which can be used to calculate such things as efficiency, which link to employee productivity.
• Qualitative Results: In addition to the metrics, Usability testing gets at why users take certain actions. This is something A/B testing is not capable of.
• Frequent Feedback: Usability testing can be set up as an ongoing piece of the process, to generate constantly updated user feedback and increase user knowledge.

As noted above, Usability tests differ from A/B tests in some significant ways:

• Usability testing can be much more time consuming and cost prohibitive than A/B testing, requiring a facilitator to be available for each test conducted.
• Usability testing can’t run on “set it and forget it”; users need to be brought in for each test and a test facilitator needs to be actively involved.

At its best, usability testing begins early on in the project, and is conducted regularly throughout all phases, providing team members with continually updated information. The qualitative data of seeing through a user’s eyes can save time in the overall product direction and creation process. In short, there is no better way to learn than through the user’s eyes.

Choosing the Right Test Every Time

A/B testing and Usability testing work very effectively hand in hand. By conducting regular usability tests, your knowledge will continue to grow over the development of your product. Once the product is at the right place in development, A/B testing shines; options for wording, buttons and designs can be tested on larger groups over a longer duration to see which performs better. Follow the A/B testing with another round of usability testing to understand the “why” and your product will be in good shape.

Furthermore, by understanding the context that usability testing adds, and determining the most effective design choices through A/B testing, you also begin to see the bigger picture: the emergence of a cohesive and effective product – one that works as the user expects, because you understand those expectations. And when your product succeeds on that level, you can begin to achieve the larger goals that determine its overall success.

Unfortunately, there isn’t always time or budget to effectively do both tests. Often, you’ll need to determine which test is most appropriate for the current state of the project. Checking the project goals is a good starting point; from there, consider what questions you have. If you’re seeking answers to “how many” more often than “why” to meet these goals, A/B testing is the way to go, and vice versa. Here is an infographic to help you choose the “right” test:

How do you determine which test to use?