To be honest, I think – at least when it comes to usability testing – all questions lead to biased answers.
As I was just telling one of our new usability engineers the other day, a usability test is not an interview. They’re supposed to be task-based. And what that means is that, other than prepping the user for think-aloud and giving them a scenario, you need never say another word. It’s all about the user completing the task and vocalizing what they are doing and thinking. The perfect test is one where you just sit there and take notes.
Now, though I do get these every once in awhile, imperfect tests are much more common. Users only rarely will make it that easy for me. Most of the time, I have to work a little bit harder to earn my pay.
Indeed, there are no shortage of times when you have to say something. Most often, the user may simply fail to do the think-aloud. I think all usability engineers are familiar with the classic “What are you thinking?” just for those situations. Variations on this include, “What just happened?” “What’s going on?” “Is that what you expected?” and so on. And, once users do start talking, a simple “un-huh” is usually enough to keep them going.
Even if users are doing the think-aloud, though, sometimes that’s not enough. Frequently, they may leave thoughts hanging. They might say, “That’s a nice …” or “I don’t know if …” or “How did that …?” Because we humans hate to leave anything incomplete, simply repeating what they just said will usually prompt them to complete the thought.
You can use a similar trick to get users to elaborate on vague comments like, “That’s not good,” or “I don’t like that.” Simply repeating their phrase will get them to almost magically add the “why” (and in a way that sounds a lot less accusatory than simply asking them why).
All in all, the less questions the better. And, if you do have to throw in a few, make it so they don't even sound like questions.
A lot of this advice came from an excellent talk Judy Ramey (at the University of Washington) gave at a UXPA conference many years ago