A wonderful interface solving the wrong problem will fail. (Jakob Nielsen)

In other words, there is a big difference between usability and utility. 

Say you test a new system, and it performs well. It’s then released … and sinks like a stone. What just happened?

Sounds like something that worked just fine, but didn’t actually solve any user needs. In other words, it’s usable, but useless at the same time.

So, should usability engineers be testing for both conditions? Isn’t our job merely a matter of the former, and not the latter?

Well, if you really want to add some value to what you do, though you can concentrate on usability, you need to keep your antennae out for utility as well. An obvious way to go about that is to ask upfront. 

The SUS questionnaire, with its first statement being “I think that I would like to use this system frequently,” is perfect for that. Just be sure to follow up. Another thing you can do is to just ask straight out in the debrief. A question like, “Is this something you would use?” should work just fine. 

One more thing to consider is to include self-directed tasks. In other words, instead of a super-specific task that asks the user to, say, find what date check 1338 cleared, you could start out by asking them if they use checks. If they say no, you now have some information about the utility of that particular feature. You can also, of course, still ask them to complete the task (“for the sake of this exercise,” I usually say). An additional benefit of this approach is that it keeps users task-oriented, but also gets them to start thinking about the value of what you’re asking them to do. 

Now, is there a way can you stop such problems from happening in the first place? Well, it’s really not part of a usability test, but if you can get involved upfront – doing ethnography, in-depth interviews, focus groups – these are obvious places to address the utility question and to get at users’ true unmet needs.

Heck, why go to all the time, expense, and trouble of solving a problem that doesn’t even exist? Surely, there are more than enough problems out there that do.

More from Jakob:

• "The fact that surprising results happen is exactly why we conduct actual research."

There is no such thing as a “user error.” (Anonymous)

Boy, do some users love to blame themselves. You know the type … “Oh, I’m not very good with computers.” “You know, my husband would be a lot better at this.” “Man, that was dumb of me.” In my informal taxonomy of user types, I call this type the “Charlie Brown.” 

I usually just tell them something along the lines of, “Oh, no, not at all.” If they persist, or if it really seems to be an issue, I do the traditional spiel about “this is not a test of you.”

If that doesn’t seem to be getting through, though, I usually follow that up with:

“This is a test of the system. You can do no wrong here today. If there’s a problem, it’s a problem with the website [or app or whatever], and I want to hear about it.  That way, we can fix it up, and it won’t be a problem for others.”

I might also talk about how they were recruited just for this test and are the perfect person for it, that the system was designed for someone just like them, and that I’ve got 10 other people coming in that week who are exactly like them. I don’t like to do it, but if the user is really struggling with this issue, I might even go so far as to mention that other users had exactly the same problem.

Now, on the other hand, there are also some team members who love to blame the users as well. With them, my approach is a little bit different. ;^)  I might start out by cocking my head, frowning, and giving them the eye. If that doesn’t work, I usually point out that this person is a customer. Now, I might also sympathize a little with them by confessing that the user was difficult for me as a facilitator and that they were definitely on one side of the sophistication scale. That said, I also try to firmly get across the fact that this is a real user and needs to be addressed somehow in the design.

And if that doesn’t work, I have no hesitation about reading that observer the riot act. That usually involves reiterating that the “user is not you,” there are many different user types, empathy is a sure sign of a good designer, and – finally – these “idiots” also just so happen to be paying your salary.