People often test to decide which color drapes are best, only to learn that they forgot to put windows in the room. (Steve Krug)

Over the years, I’ve actually found that this is more of a problem with more experienced clients and observers. Newer ones are often happy to just be there. They often have no idea what to expect from a test. More experienced folks, however, often go into testing with preconceived notions of what the problem is and what testing is likely to find.

Here’s how it typically goes … One of these more experienced people will come to me and ask for a usability test to get at some minor detail in some design their team has been working on. To them, everything else is perfect, but there just so happens to be this niggling little detail that the design team just couldn’t get closure on. That's all it is ...

So, the first thing I normally do is try to get the client to step back a little bit. I usually just ask what the design is for, what it lets the user do, how it’s supposed to work.

I then get the client to agree to making the test more task-based. And that’s what usability tests do best – getting real users to sit down in front of a system and try to do real tasks. Now, that will get at the particular thing the client is interested in, but it will also get them feedback on the rest of the design, on the user’s whole experience, on things that might never have even occurred to the design team.

The final thing I have to do is to get the client to have an open mind.  ;^)

It really is a little ironic that I’m the one who always expects to be surprised. I mean, I’ve been doing this for almost 30 years, and spend at least 100 hours every year one-on-one with users. I should be the last person to be surprised.

That actually, though, may be the favorite part of my job. I’m always learning something new. In fact, if that wasn’t the case, and everything was predictable, I probably would have left the profession long ago. Maybe it’s because I’m basically a scientist at heart, but I always approach any test with an open mind. I really can’t wait to see what turns up. Surprise me, users!

Couldn't have said it better myself

Steve also said:

• "Happy talk must die."
• "The point of testing is not to prove or disprove something. It’s to inform your judgement."
• "Instructions must die."
• "It doesn’t matter how many times I have to click, as long as each click is a mindless, unambiguous choice." 
• "When fixing problems, always do the least that you can do."