Saturday, February 28, 2015

What people say, what people do, and what they say they do are entirely different things. (Margaret Mead)

I think the perfect example of this is when, at the end of a test, I ask the user how things went. Invariably, they’ll say it went “pretty good.” And they’ll say this no matter whether they aced it, flunked every task I gave them, or – as most commonly happens – fell somewhere in the middle.

I’ll typically also have observers who will hone in on that particular part of the test, citing it as evidence that the users “liked it,” the test went well, and nothing really needs to be changed. This usually only happens for brand-new observers however. I’ve invoked this particular quote so often that I’m pretty sure that anyone who’s sat in on more than a few of my tests has already committed it to memory.

I do also explain, though, that what’s going on has a lot to do with social niceties and – to a lesser degree – the limitations of human memory. I then try to focus them on what actually happened, reiterating that usability tests are task-based.

I also make the point that a task-based test is a real reflection of how people actually use websites. So, yes, Mr. Art Director, when the user says that they liked the colors? They’re really just being nice (or are not sure what else to say).

I’ve also used Dr. Mead’s quote when I need to emphasize the limitations of surveys – including self-reporting, self-selection, and retrospection. And, though I run focus groups myself, I also use the quote to make sure that my team realizes what focus groups are – and are not – good for. Finally, I also use it to make a plug for ethnography, a field that owes a lot to Margaret Mead.

Unfortunately, I cannot use that quote when it comes to web analytics. In fact, I once had a senior exec take a shot at usability testing by firing that quote right back at me. After all, web analytics really are what people do.

After fumbling around a good bit, I was able to make the point that this is one time where the “say” part of the equation is pretty darn important. In a usability test, what the user says while he completes his task lets us know what’s going on in his brain – he’s looking for x when we call it y, he’s looking at the top of the page when we put the button at the bottom, he thinks he’s finished but there’s one more step he needs to take. And analytics might not tell you any of that. And that's why this particular quote is still one of the best ones in my armory.


Unfortunately, this book has nothing to do with usability

Monday, January 5, 2015

People don’t want to buy a quarter-inch drill. They want a quarter-inch hole. (Theodore Levitt)

I am surrounded by techies. And it’s not just the developers I’m talking about. It seems like every one is pretty technical these days – graphic designers with their Photoshop, writers with their content management systems, usability engineers with their eye-trackers, elementary school students with their iPads …

So, here’s the thing about real techies … They really want that quarter-inch drill! Yeah, they might make a quarter-inch hole at some point … But hey, look at this cool drill! It’s made of helical-cut steel, and its gears are heat-treated steel as well. Plus, it’s 510 Watts, and it’s no-load speed goes all the way up to 4,000 RPM! And that’s not to mention the keyed chuck it’s got on it too.

So, does it drill a quarter-inch hole? Well, with a twist bit it does. Heck, it’ll drill a three-quarter-inch hole if you’re using a spade bit.

So, do I want to want to drill a three-quarter-inch hole? What’s a twist bit? How am I supposed to know? Where am I?

I actually have no idea what I just said (except for those last few questions). It’s not that my father – who had a woodshop, as did his father before him – didn’t try to make me understand. I just didn’t care. Sure, the racetracks and train sets they both built were particularly awesome and I couldn’t wait to play with them. It’s just that I wanted – in this particular situation at least – to play. I didn’t care about the thing and how it was made. I just wanted to use it.

Even today, though, we are still often forced to care about how something was made. It’s usually not that we’re going to make it ourselves, but that its poor design forces us to think about its construction. It is never as seamless or intuitively obvious as it claims to be. And that’s what gets between us and our goal, because our goal will always be the train set and not the drill (unless, of course, you’re like all the other males in my family).

Really, this is all just about the old Steve Krug saw, “Don’t make me think!” I don’t really want to think about my banking app, or that travel aggregator site, or “Autosense Technology that drives most screws flush on the first try.” I simply want to make that transfer, or book that room, or just make that quarter-inch hole.

Theodore Levitt (Harvard Business School professor,
longtime editor of Harvard Business Review, and well-known author).

Tuesday, December 9, 2014

Criticism is a form of optimism. Only silence is pessimistic. (Carlos Fuentes)

I once heard one of my colleagues describe herself as a “professional scab-picker.” And that, sometimes, is a pretty apt description of what we do. Or at least a good representation of how some people view us, if nothing else.

Now, I like to think that that last group probably just didn’t work with a more seasoned usability engineer. Let me tell you, letting someone know their baby is ugly takes some real skills. Not only do you have to report some positive results as well, you have to be convincing with your arguments, offer some possible solutions, have enough emotional intelligence to know how your report will be received, and just let some things slide.

Actually, sometimes this has more to do with the team receiving the criticism than the one giving it. In particular, I’ve noticed that it’s the more seasoned, experienced teams who can’t wait to get something into the lab, come what may. It’s the ones newer to usability that might have a tougher time – whose skin is a little thinner, who tend to be a little more defensive. Ironically, it’s also typically the less experienced team that has more to fix, and the more experienced one much less. Ah well.

Now, I have been in some situations where I have, effectively, been “silent.” These typically arise when there is such a mismatch between a team’s capabilities and any form of usability that it simply just isn’t worth it. Are you surprised that there are actually places in this day and age where this might happen?

Well, it’s true. I just so happened to work on one just a little while ago. It was basically an internal team, developing an internal tool, for an internal client. Everyone was very heavy IT, and the atmosphere actually gave me a very heady feel of the 1990s. They had heard of me somehow or other, and I made some pretty basic suggestions as delicately as I could over the phone on our first contact, explaining everything in detail as I went. I repeated that approach again in another phone call, then a conference call, and then another …

At that time, it became pretty obvious that no one was going to actually act on any of those suggestions, so I eventually retired from the field. No reason dying on my sword for this one.


Carlos Fuentes, Mexican novelist. I’m pretty sure he never uttered the word “usability” in his life, but any author who says the first thing he thinks about when he begins is a book is “Who am I writing for?” might actually not have a hard time understanding user-centered design.

Wednesday, November 12, 2014

It is easy to prove something is not usable with small sample sizes. It is hard to show that something is usable with small sample sizes. (Jim Lewis)

Boy, do I get this one a lot these days. Interestingly, though, I never got it at the beginning of my career, when I worked mostly with techies. Nowadays, though, the web seems to have been pretty much taken over by marketeers – and, boy, does this resonate with them.

And that’s because the typical marketeer is very data-driven. Web analytics, VOC (voice of the customer), CSAT (customer satisfaction surveys), NPS (net promoter score), KPIs (key performance indicators) … Throw about a dozen brightly-colored graphs and charts on a screen and lots and lots of numbers, then call it a “dashboard,” and these people are in heaven.

Now, a usability report does not look a lot like a dashboard. So, there’s typically a couple of points I like to make with this group.

First, this is qualitative research. They’re usually familiar with focus groups, so if you can make this connection, you’re already halfway there.

Second, like all good qualitative research, usability testing gets at the “why” of the issue. Yes, it’s nice to know that A/B testing showed a definite preference for B, but wouldn’t it be nice to know why that is? Maybe we can use a similar strategy for our next design. Could work.

Third, this kind of research is not about preferences. What it is is about bugs. Now, these are not the broken links, infinite loops, etc. marketeers may associate with QA. To the user, though, they might as well be. These are the obvious things – the mislabeled buttons, the hidden links, the missing help – that trip lots of people up. In fact, at this point, I usually bring up the old story of the bunched-up carpet in the hallway, and ask the marketeers how many people they would need to see trip over it before they would fix it.

Finally, this kind of research goes before launch. Marketeers really can’t say that about their nice, shiny statistics. In fact, once you start counting, whatever it is you want to count will be out there for all the world to see. Wouldn’t it be nice to get a little feedback on what you might expect before everyone else sees it?


Jim Lewis has been doing great work at IBM since 1981! He has a PhD and is the author of Quantifying the User Experience: Practical Statistics for User Research, with Jeff Sauro.

Friday, October 24, 2014

Never take a fence down until you know why it was put up. (Robert Frost)

I’m a strong believer in the law of unintended consequences. I think it fits my basically pessimistic nature. 

So, while the marketeers and execs and genius designers are all ready to – oh, I don’t know – turn every word into an icon or have all navigation be done by gestures, I’m usually the one who has to rein things in. Honestly, sometimes I feel like the ballast that keeps the balloon from wandering off into the troposphere. Unfortunately, all of that gives me a reputation for being rather conservative, which I do not relish. 

Now, this is not just a matter of my saying “no” to everything. What I like to do, instead, is ask these questions:
  1. Do we know already if it’s working or not working, or are we merely guessing?
  2. Do we know if the new idea will work as well? What would we be basing that on?
  3. Would you like me to help you answer that second question?

It’s that third question where I really like to position myself. Basically, I’m not shooting down their idea so much as simply getting them some feedback on it. Yes, I do have my opinions on the topic. And, yes, those opinions are typically based on spending a lot of time with real users … and seeing things work … or not work. But wouldn’t it be great if we could get this out of the realm of total conjecture and seat-of-the-pants intuition and see whether something will actually fly or not? 

My Dad, the electrical engineer, took one of Frost’s poetry classes 
while at Dartmouth (the buildings in the background). 
It was his favorite class.

Tuesday, September 23, 2014

If you can’t explain it simply, you don’t understand it well enough. (Albert Einstein)

I got my start as a tech writer. One thing I began to notice very early on – and be rather concerned about – was when I had to writes pages and pages explaining a particular feature. Was all this verbiage really necessary? Could the way that feature operates be made a little simpler?

At some point, I started going back to my developers and asking them about it. Sometimes, they ignored me. Sometimes, they took me seriously. And, sometimes, they actually tried to come up with something simpler and more elegant.

I then made the rather monumental decision of coming up with some ideas on my own and presenting them. These led to some pretty good discussions – and with some of my ideas actually getting implemented. Hmm, I said to myself, this is pretty cool!

In fact, a number of the developers I worked with started to expect something from me in the way of a solution. I guess it was just easier than coming up with something on their own. And that’s how I got started in this business.

So, I don’t know if I’m in total agreement with Einstein here. Yes, I do get his point. Sometimes, though, complex things need complex explanations. The question we have to ask ourselves, as UX professionals, is whether that complex thing needs to be quite as complex as it is. If we can make it simpler, not only will it be simpler to explain, but it will be easier for the user to understand and use. And isn’t that the real goal here?


Tuesday, September 2, 2014

The most savage controversies are about those issues for which there is no clear evidence either way. (Bertrand Russell)

Some days, it seems these are the only issues I deal with.  :^(  What’s interesting, though, is that I never seem to come up with these on my own.  

I’m pretty evidence-based. I think it’s rather hard to be a usability engineer for 25 years and not be. More than that, though, it’s just the way I operate. I like nothing more than having my biases disconfirmed. That means I actually learned something that day.

As for the people I work with? Maybe not so much. The execs, the marketeers, and even the interaction designers and information architects on my team don’t always necessarily seem to do things that same way.  

How do they work instead? I think some of them might call themselves “passionate.” Now, that’s all well and good. Sometimes, though, that “passion” can strike others as simply “loudest voice wins.”

Now, I do have to say, that these people genuinely are passionate. They’re also typically pretty darn smart, and know their stuff as well. Finally, they do respect actual evidence. They may just not have any at hand – not that that’ll stop them though.  ;^)

Now, I’m passionate, and smart, and know my stuff too. The role I typically carve out for myself, though, is all that stuff, but subbing out the “passion” and throwing “evidence” in there instead (okay, I tend to be passionate about my evidence). And it’s really amazing how well that works, how that can stop all the conjecture and debate and “savagery” in its tracks.

So, here’s how I typically do that:
  1. Interrupt with some real data that is directly applicable
  2. Interrupt with some real data that is applicable but maybe a little less directly
  3. Start counting on the team to ask me if there’s any data
  4. Have no fear of saying, “I don’t know”
  5. Offer, though, to find out for them

Bertrand Arthur William Russell, 3rd Earl Russell, OM, FRS