Expert usability review vs. usability testing

By Inviqa

We're often asked which method is best: usability testing or expert usability reviews. So let's explore the key differences between these approaches, and identify a winner. 

The difference between user testing and reviews

An expert usability review is where a usability specialist inspects a website to identify potential usability problems.

Usability testing, on the other hand, incites people from your target audience to evaluate your site whilst performing defined tasks.

The two usability approaches are similar in that they both:  

  • Find and prioritise usability problems
  • Evaluate designs in the context of tasks

But do they find the same problems?

Generally speaking, expert usability reviews tend to find high level breaches of design rules and consistency.

Meanwhile, usability testing is better at finding issues related to special domain knowledge and task flows.

If you were to carry out an expert usability review ahead of usability testing, you'd be able to see how the issues identified in your review compare with the real experiences of your audience.

This will give you a good feel for how expert usability reviews hold up against usability testing.

The limitations of usability reviews

Often, expert reviews will:

  • Miss usability issues that arise during usability testing
  • Identify some issues that usability testing does not
  • Report false alarms (i.e. not real issues)

And that's the problem:

If you solely rely on a usability review, you'll identify usability issues. But you could easily overlook the more important usability issues that would surface during testing with real users.

Consider the case of a health website aimed specifically at new mothers.

An expert usability review found plenty of usability issues, but missed a key issue that caused participants to make comments such as 'That's just so insensitive!' and 'I wouldn't use this site anymore'.

And all because of a seemingly benign bit of banner advertising for a local gym placed next to an article about breast feeding. A reviewer would have difficulty predicting the indignant reaction of new mothers who felt the site was rubbing their nose in it (after all, there would be little chance of them going anywhere near a gym with a newborn).

Is there overlap between these usability approaches?

Naturally, there is some overlap in the problems found in both expert usability reviews and usability testing.

Research conducted by Fu, Salvendy, and Turley in 2002 estimated this to be around 41%. Not as high as you might expect.

Interestingly, expert usability reviews tend to find more problems compared to usability testing. But it's often a case of quality over quantity.

Usability reviews are valuable...

Expert usability reviews are more common than usability testing simply because they're cheaper and quicker to do. Typically, it'll only take a few days to inspect a site and write a report and because you don't need to hire labs nor recruit users, costs are significantly cheaper.

Because of this, they're often preferred to usability testing, especially when time and budget are squeezed.

There are, however, some pitfalls. For a start, a lot depends on the reviewer's level of experience. No two usability experts will find exactly the same issues so the method is also open to practitioner variation.

But the biggest problem is simply that they sometimes miss the real problems that cause users to fail tasks.

This can be especially true when the target audience has a particular skill set. For example, an accountant may be better placed to diagnose certain types of issues related to online accountancy tasks than a non-accountant reviewer.

...But usability testing is invaluable

Usability testing takes more time to plan and organise and is more expensive too; recruiting and incentivising people from a target audience can be costly. But it's definitely worth the effort! The results give a true picture of the real problems people encounter since they're derived from real users in the first place.

It's inconceivable how sites go live without checking to see whether they're actually usable; it's like building a car but not test driving it. 

Another advantage of usability testing is that there's less conjecture, and feedback comes straight from the horse's mouth.

Videos of user sessions, in particular, can provide a powerful persuasion tool for reluctant stakeholders. It's hard to argue with recordings of real, frustrated users.

Always test with real users

Both expert usability reviews and usability testing have their time and place. In practice, people often use expert usability reviews early on to straighten up their design in preparation for usability testing. (Indeed this is critical for accessibility testing, since it's a waste of time to recruit and test users who can't even get into your site in the first place.)

Whilst an expert usability review is better than nothing, it's still vital to put your website in front of users as early as possible too, rather than leaving it too late.

Key elements, such as site structure and navigation, should be assessed with users at the earliest opportunity. Otherwise you risk having your website completely overhauled at a late stage.

Ultimately, the key is to appreciate that expert usability reviews and usability testing are different beasts. The most effective approach is to try and integrate both techniques. But always test with real users. There's no substitute!