If you don’t have the right users, you don’t have the right data. (Dana Chisnell)

There’s a reason why the usability engineer is asking you to review that incredibly long screener.  It’s to make sure you get the right person in that chair in the lab at 9:00 next Tuesday.  

Have you ever seen what happens when that doesn’t take place?  Believe me, it isn’t pretty.  A test of a CD (certificate of deposit) product on someone who thinks you’re talking about music?  Maybe not such a good idea.  A test of a mobile app with someone who just bought an iPhone, but hasn’t really used it yet?  You might want to think about that one.  A standard, think-aloud test with a participant who just isn’t comfortable thinking out loud?  Perhaps we ought to reconsider.

Of course, the problems don’t have to be so blatant, and can be a lot more subtle – and can still cause lots of issues.  Designing a mobile app for people in their 40s and 50s, but your participants skew more toward the 20s and 30s?  Well, don’t be surprised if that tiny little button that your participants had no trouble tapping will cause some issues down the road.  Or if actual customers never use that particular gesture that your test participants (and your 20-something designers) just loved.  Or if the real users can’t read that link because the text is too small.

(That said, I sometimes have just the opposite experience, usually when I work with marketing.  I find they’re used to demographics.  So, in addition to the traditional mix of genders, ages, and so on, they typically also want to make sure we get some Midwesterners and some African-Americans females and some stay-at-home Moms in their mid-30s in a certain set of ZIP codes.  The typical joke I make here is about left-handed, green-eyed, Presbyterian Capricorns.  Sometimes they even laugh at it.) 

Actually, I’d have to up Dana one and say that if you don’t have the right users, or the right tasks, or the right system, you won’t have the right data either.  A usability test is lot like a soufflé recipe.  You miss one of the ingredients, and the thing’s going to fall flat.  

Dana