A successful test is a test that uncovers many problems. (Rolf Molich)

I often get asked in the elevator or the break room or the wash room how “that test is going.” Even after all the years, I’m afraid I have to admit that I never know quite what to say.

If it’s someone on the project team, they basically want to know that everything’s hunky-dory and that I have essentially uncovered no problems [yeah, right]. If it’s another usability engineer, I might delightedly share that the system’s a piece of crap, or that my 1:00 appeared to have had a few drinks with lunch, or that that method they suggested is working well. If it’s a developer, they may simply want to know that the prototype worked.

For me, though, what I’m really looking for is consistency. What I really like is when the first three or four or five users all have trouble with widget X, or label Y, or screen Z. Even though we are dealing with pretty small numbers, that consistency makes me feel much more confident.

(Now, there are never any shortage of problems that only one or two users might uncover. What I do in that situation is take a closer look at these issues and see if the user has uncovered something valuable or if they are simply an outlier. That’s always a judgment call, though, unfortunately.)

My ability to tie a problem quickly, directly, and clearly to a specific field or interaction or missing piece of information also makes me feel good inside. Some of the hardest reports to write are those where there was obviously an issue, but what it was and what was causing it weren’t so clear. Of course, it’s always a great feeling when I’ve solved these knotty problems, but there can definitely be a fair amount of anxiety leading up to that moment.

One final thing … Yes, most of my tests do find plenty of problems.  Plenty, though, also uncover many things that work. And some even uncover plenty of good things and very few problems. And I’ve always believed that’s just as important to present as when there’s nothing but issues.

Unlike a lot of usability engineers, I do like to emphasize the positive. I just find my report recipients are a lot more open to what I have to say if it’s not all negative. So, maybe Rolf’s quote should read, “A successful test is a test that uncovers many things,” or “A successful test is a test that uncovers a lot of stuff,” or … Ah, forget it, it’s still a pretty darn good quote just as is.

 Also from Rolf:

• "God is on our side, but we can help him."
• "Help doesn't."
• "Even developers have feelings."