Those are definitely my thoughts, though, whenever I hear someone put forth the traditional, “Where do you get these idiots from?” (or something along those lines). How I actually do respond is to point out these are our users, we used a 10-page screener and paid a recruiting agency $1000 to get ahold of them, and that not everyone out there is as smart and tech-savvy as you guys.
So, that usually takes care of the “smart” part. As for the “doing their best,” we sometimes do have users who are just there for the money, but that’s extremely rare. It’s usually totally obvious to anyone observing that 99 out of 100 users are taking things seriously and are genuinely engaged.
Now, as for “context” … Hopefully, the design team had some exposure to that beforehand. Personas, journey maps, and all that other great upfront research can give the team some real feel for their users – what they do and don’t know, what they like and don’t like, what their goals and fears are – and how to design something just for them.
Even if there has been that exposure, though, I try to push testing an excellent way to get even more context. Even the best upfront research can be incomplete, or neglected, or misapplied. Testing, though, is the chance to really check things out, to get that final word. The more sophisticated teams I work with have no problems understanding that, and often see testing in this regard as simply fine-tuning.
It’s those teams who don’t do any up-front work, and who can be totally blind-sided by things that happen in the lab, that I really worry about. Hopefully, though, these teams can use that experience to learn to emphasize with their users a little more – heck, maybe even do a little of that up-front research and avoid those uncomfortable situations in the first place.
Just in case you were wondering what a HIPPO is