What usability testing doesn't tell you

Most people in UX have a pretty good grasp on what usability testing can tell us. But it's important to maintain an equally good grasp on what it can't tell us.

Let's say you have an idea for a new mobile application that lets you work with email in some new way, and you're thinking about how users might organize their email. Maybe you arrive at two options: organizing by sorting, and organizing by applying tags. You're not sure which to implement as the primary approach, so you do some usability testing.

The problem is that even if you're able to mock up a test application that presents your tagging idea, any population you find with email experience will almost certainly already have experience with using sorting to organize email. At the very least, the email systems they've used have employed sorting as the primary organizational method.

Now, I don't know (or at the moment, particularly care) whether sorting or tagging works better for organizing emails. The point is that the two things represent different underlying concepts of "how to organize". Usability testing is not good at testing underlying concepts. Established concepts are familiar, and although a new concept may have great advantages, it's often hard to understand them right away, particularly in the very abbreviated time usually alloted to a participant in a usability test. So the established concept is very likely to appear more usable in the test.

Usability testing doesn't inevitably reinforce the status quo, but it's something to watch out for.