I once put this quote on a report I put together for a client who was really into counting. It may not have been the best idea, but it did get their attention. And several years later, it looks like they may actually have found a place at their table for qualitative research.
I use a couple of arguments when I have to stand up for the legitimacy of qualitative research with more numeric types (those who are really into surveys or analytics, for example). First, I simply point out that we’re both simply creating some sort of feedback loop. The one real advantage of usability testing is that we can get that feedback before something launches.
At this point, I often have to make a distinction between usability testing and QA. What I typically stress here is that usability testing can happen at any point in the cycle (from pieces of paper to production systems) and that it has a much broader focus (not just, “is it broken?”).
Next, I usually point to qualitative work that they may be familiar with. If they’re marketing types, that usually means focus groups. (Though usability engineers may have trouble with these, marketeers typically do appreciate this method.)
I then make the point that there is often a real trade-off between numbers and richness of data. Methods that emphasize the former (web analytics, say) tend to give you a real good feel for what’s happening, but often don’t tell you why it’s happening.
Finally, I like to point to the famous graph that Jakob Nielsen (with help from Bob Virzi) came up with:
I also typically mention – in an offhand way – that this is a perfect example of an asymptotic curve (and ask them where they think the asymptote would be). It’s usually at this point, when they’re so bowled over with my brilliance, that I can get them to agree to anything. (That’s a joke, by the way.)
Einstein also said:
Steve Krug and Caroline Jarrett put on an excellent session at UPA 2012 called “’...but the light bulb has to want to change’: Why do the most serious usability problems we uncover often go unfixed?”
To me, this has always been the proverbial elephant in the room. Everyone knows usability engineers do great work. How often, though, is their work actually acted upon? How often do things fall through the cracks? How often do project teams pay only lip service to usability?
In a previous life, I once had a little time on my hands … and decided to find out. I went through a year’s worth of usability reports, then tallied up what percentage of issues our different clients actually fixed. As it turns out, the client that we seemed to be on the best standing with (they wanted our services constantly, observed sessions religiously, and seemed to “get” usability) fixed the fewest issues. And a client who we had wrestled with on what seemed like everything actually fixed the most! I’m really not sure what was behind all this, but it certainly was an eye-opener.
Steve and Caroline (in a survey that they ran on over a hundred usability engineers) found that the major culprits were capacity and politics. Other issues included changes to business processes, technology, holding the fix for a future redesign, and legal. Some ideas for mitigating these issues include:
- Prioritizing (for example, putting easy fixes that have major impacts at the top of the list)
- Making testing more collaborative (getting team members to observe, debriefing with the team, etc.)
- Speaking the language of business and focusing on their priorities
- Focusing on the big problems (i.e., avoiding the temptation to list anything and everything in your report)
- Doing the least you can to fix the problem (e.g., not redoing the whole system when adding a help link will suffice)
- Equating usability bugs with any other kind of bug
- Putting fixing usability bugs in the schedule
- Celebrating fixes
I like these ideas. Heck, anything that would get around reporting the same issue over and over again would be a winner with me.
John Foster Dulles was not a usability engineer, but was Secretary of State during the Eisenhower administration. Dulles Airport, in DC, is named after him. As afar as I know, there are no airports named after famous usability engineers.
There’s a certain kind of person who loves a challenge. That beautiful kitchen island with the marble countertop? They built that. That underground sprinkler system? They put that in. That home network? They set that up.
Then there’s the rest of us. We just want the stinking printer to work … so we can get our report … and go home.
Now, here’s the rub … As usability engineers, we tend to work with the first type of person. But we usually advocate for the second. When we’re in the lab, we work with the latter, but then typically have to explain what happened to the former.
It can be quite a challenge operating as translator. Luckily, most of us are used to drawing on both sides of our brain. In fact, that’s why a lot of usability engineers come from fields like technical writing or instructional design, in my opinion.
At the same time, though, we are engineers. And there are some of us who take that part of the title very seriously. They might be stat heads who are interested in ANOVAs, Bayesian inference, and stochastic processes. Or they might just be closet developers.
The latter are the ones I worry about. For those folks, understanding and working with users can sometimes seem to take a backseat to making the eye tracker run or tweaking the prototype or coming up with some homegrown tool for this or that. I know there are true Renaissance people out there (and I know I ain’t one of them), but I sometimes wonder if usability engineers like this haven’t gone over to the other side.
Mary Beth Rettger, Directory of Usability at The Mathworks and former UPA president, has called herself a “luddite.” I don’t know if I want to go that far. I do, however, like to keep myself somewhat “pure.”
The only time I’m around non-techies these days seems to be when I’m in the lab with my users. Being able to channel them outside the lab is a lot easier if I feel I genuinely have something in common with them. So, I guess that’s why I’ll always be more on the “usability” – and less on the “engineer” – side of “usability engineer.”
Not Mary Beth Rettger
I’ve always found it ironic that usability engineers, who are so centered on “the user,” often neglect to keep their own users in mind. I’m not talking about the end users of some system or website, but the usability engineer’s own clients, the people who receive the reports and pay the bills.
This problem seems to be most evident in usability reports. I’ve seen a number over the years that just aren’t very friendly. And so has Rolf. In fact, one of his main interests is helping usability engineers create better, more usable reports.
So, what are Rolf and I griping about? Now, this is by no means something that is the problem that it was years ago, and probably only applies to a small minority of usability engineers these days. I guess I put it all down to two things…
One is that we are engineers, right? So, it’s an engineering report then, correct? And we all know that engineering reports are long, and boring, and have plenty of data and tables and appendices.
The second is probably simply a hangover from academia. God knows there are an awful lot of PhDs in this field. And we all know what journal articles are like, right (and how long, boring, and full of data and tables and appendices they are)?
Well, as it turns out, our actual audiences cannot typically get as jazzed up about the traditional usability report as we can. Our audiences are usually not other usability engineers or editors of academic journals. Our audiences are typically information architects, and interaction designers, and marketers, and execs, and graphic designers, and content specialists, and – can you imagine – even developers!
There are plenty of things we can do to make our reports more usable for our actual readers, but one thing we can do to help developers (and any person, really, who had input into the look and feel of what we are testing) is simply to be nice. Now, there are several ways to do that.
One that I have fallen in love with over the years is to simply include some positive findings in my report. I have heard other usability engineers balk at that, saying that we’re all adults here, that we’re not getting paid to stroke egos, etc., etc. I find, though, that responding well to positive reinforcement is simply the way that most humans work – and that includes developers as well.
There are three ways to design a website or piece of software. One, you can just code it up and throw it out there. Or, before you throw it out there, you can run it past some real users. Or, if you really want to do this user-centered design thing, you can do some user research upfront.
That last piece, in my mind, is what really separates the men from the boys (the women from the girls, the adults from the children – you know what I mean). It allows you to bake in usability from the very beginning. That said, I’m often surprised at how little it’s done.
So, what does that actually involve? It might be as simple as interviews, or as complex as a field or diary study, or something in between, like a focus group. It’s what you do with that data, though, that’s really crucial.
And that’s where personas come in. Upfront research can generate a ton of information. What the researcher’s real task becomes is how to take that fire hose of data and turn it into a nice, quick, refreshing drink. And, for that, personas really can’t be beat.
I think we all know what personas are by now. You take all that information you uncovered, then distill it into several user types. What really makes the persona, though, is how we can turn each one into a real person – with a name, a picture, a place where they live, a job they go to, a family …
Humans are social animals. They also love a good story. What personas do is take those two very important human characteristics and turn them to your advantage. Instead of designing for a complete abstraction (or, worse, with no one or yourself in mind), how much easier it can be to keep the user in mind by thinking of Phyllis with her fear of technology, or Bill with his impatience, or Julie who is trying to juggle family and job and caring for her elderly parents.
Over the years, I have gotten some resistance to using personas. It’s often a little too touchy-feely for computer science or MBA types. What’s gratifying to see, though, is how readily personas are adapted, when given just half a chance. It’s so nice when developers start talking about Ben or Carol or Leah. To me, though, that’s just human nature.
Kim Goodwin has a pretty good background in personas, having worked for Alan Cooper, the creator of personas, for 12 years. She’s also the author of Designing for the Digital Age.
There’s a reason why the usability engineer is asking you to review that incredibly long screener. It’s to make sure you get the right person in that chair in the lab at 9:00 next Tuesday.
Have you ever seen what happens when that doesn’t take place? Believe me, it isn’t pretty. A test of a CD (certificate of deposit) product on someone who thinks you’re talking about music? Maybe not such a good idea. A test of a mobile app with someone who just bought an iPhone, but hasn’t really used it yet? You might want to think about that one. A standard, think-aloud test with a participant who just isn’t comfortable thinking out loud? Perhaps we ought to reconsider.
Of course, the problems don’t have to be so blatant, and can be a lot more subtle – and can still cause lots of issues. Designing a mobile app for people in their 40s and 50s, but your participants skew more toward the 20s and 30s? Well, don’t be surprised if that tiny little button that your participants had no trouble tapping will cause some issues down the road. Or if actual customers never use that particular gesture that your test participants (and your 20-something designers) just loved. Or if the real users can’t read that link because the text is too small.
(That said, I sometimes have just the opposite experience, usually when I work with marketing. I find they’re used to demographics. So, in addition to the traditional mix of genders, ages, and so on, they typically also want to make sure we get some Midwesterners and some African-Americans females and some stay-at-home Moms in their mid-30s in a certain set of ZIP codes. The typical joke I make here is about left-handed, green-eyed, Presbyterian Capricorns. Sometimes they even laugh at it.)
Actually, I’d have to up Dana one and say that if you don’t have the right users, or the right tasks, or the right system, you won’t have the right data either. A usability test is lot like a soufflé recipe. You miss one of the ingredients, and the thing’s going to fall flat.
Dana
I’ve been doing this usability stuff for a shockingly long time. In the beginning, we had only one enemy to blame for everything – the techy. It’s who Alan Cooper was talking about when he titled his 1999 book The Inmates are Running the Asylum.
Back then, this approach made a ton of sense. Computers had originally been developed by techies for techies. I’m talking card readers, green screens, punch cards, and command-line interfaces (all of which yours truly actually has experience with – something I like to scare my younger colleagues with when we all go out to lunch).
Then along came this thing called a “PC.” All of a sudden, techies weren’t designing for techies anymore, but for secretaries and insurance salesmen and high school teachers. And these people really didn’t want to learn COBOL or Fortran to print their label or fill out their expense report or enter their class grades.
And this is where I stepped in. Well, actually, I just happened to complete graduate school at this time. But there sure were plenty of opportunities out there for people like me to help make the world a better place (with a special emphasis on the human-computer interface, of course).
Nowadays, the blame can be spread around a little. In addition to techies, we’ve also got marketeers, bean counters, art directors, senior execs, and all sorts of sundry corporate types getting between the users and what they want to do.
At the same time, though, I have to admit that there is a lot less blame to go around. Everybody, it seems, appears to get usability and user experience in a way that no one did twenty-five years ago. Even the poor developer.
With all that said, I still think The Inmates are Running the Asylum is one of the best UX books ever written. It introduced concepts like not designing for the edge case, modeling human-computer interaction on human-human interaction (what Cooper calls politeness), and using personas and scenarios. Except maybe for its tone toward developers, I’m not sure that book will ever be out of date.
Alan, not Luke
